{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "\u003ca href=\"https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/keras_mnist_tpu.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TBsuwHGAv7w4"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "pry0oCQUv7w8"
      },
      "outputs": [],
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xqLjB2cy5S7m"
      },
      "source": [
        "## MNIST on TPU (Tensor Processing Unit)\u003cbr\u003eor GPU using tf.Keras and tf.data.Dataset\n",
        "\u003ctable\u003e\u003ctr\u003e\u003ctd\u003e\u003cimg valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/keras-tensorflow-tpu300px.png\"   width=\"300\" alt=\"Keras+Tensorflow+Cloud TPU\"\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n",
        "\n",
        "\n",
        "## Overview\n",
        "\n",
        "This sample trains an \"MNIST\" handwritten digit \n",
        "recognition model on a GPU or TPU backend using a Keras\n",
        "model. Data are handled using the tf.data.Datset API. This is\n",
        "a very simple sample provided for educational purposes. Do\n",
        "not expect outstanding TPU performance on a dataset as\n",
        "small as MNIST.\n",
        "\n",
        "This notebook is hosted on GitHub. To view it in its original repository, after opening the notebook, select **File \u003e View** on GitHub.\n",
        "\n",
        "## Learning objectives\n",
        "\n",
        "In this notebook, you will learn how to:\n",
        "*   Authenticate in Colab to access Google Cloud Storage (GSC)\n",
        "*   Format and prepare a dataset using tf.data.Dataset\n",
        "*   Create convolutional and dense layers using tf.keras.Sequential\n",
        "*   Build a Keras classifier with softmax, cross-entropy, and the adam optimizer\n",
        "*   Run training and validation in Keras using Cloud TPU\n",
        "*   Export a model for serving from ML Engine\n",
        "*   Deploy a trained model to ML Engine\n",
        "*  Test predictions on a deployed model\n",
        "\n",
        "## Instructions\n",
        "\n",
        "\u003ch3\u003e\u003ca href=\"https://cloud.google.com/gpu/\"\u003e\u003cimg valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/gpu-hexagon.png\" width=\"50\"\u003e\u003c/a\u003e  \u0026nbsp;\u0026nbsp;Train on GPU or TPU\u0026nbsp;\u0026nbsp; \u003ca href=\"https://cloud.google.com/tpu/\"\u003e\u003cimg valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/tpu-hexagon.png\" width=\"50\"\u003e\u003c/a\u003e\u003c/h3\u003e\n",
        "\n",
        "  1. Select a GPU or TPU backend (Runtime \u003e Change runtime type) \n",
        "  1. Runtime \u003e Run All \u003cbr/\u003e(Watch out: the \"Colab-only auth\" cell requires user input. \u003cbr/\u003eThe \"Deploy\" part at the end requires cloud project and bucket configuration.)\n",
        "\n",
        "\u003ch3\u003e\u003ca href=\"https://cloud.google.com/ml-engine/\"\u003e\u003cimg valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/mlengine-hexagon.png\" width=\"50\"\u003e\u003c/a\u003e  \u0026nbsp;\u0026nbsp;Deploy to AI Platform\u003c/h3\u003e\n",
        "At the bottom of this notebook you can deploy your trained model to AI Platform for a serverless, autoscaled, REST API experience. You will need a Google Cloud project and a GCS (Google Cloud Storage) bucket for this last part.\n",
        "\n",
        "TPUs are located in Google Cloud, for optimal performance, they read data directly from Google Cloud Storage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qpiJj8ym0v0-"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "colab_type": "code",
        "id": "AoilhmYe1b5t",
        "outputId": "62f7a9fe-5e4f-469f-93e8-6cb3dd9e5970"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Tensorflow version 2.2.0\n"
          ]
        }
      ],
      "source": [
        "import os, re, time, json\n",
        "import PIL.Image, PIL.ImageFont, PIL.ImageDraw\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from matplotlib import pyplot as plt\n",
        "print(\"Tensorflow version \" + tf.__version__)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "qhdz68Xm3Z4Z"
      },
      "outputs": [],
      "source": [
        "#@title visualization utilities [RUN ME]\n",
        "\"\"\"\n",
        "This cell contains helper functions used for visualization\n",
        "and downloads only. You can skip reading it. There is very\n",
        "little useful Keras/Tensorflow code here.\n",
        "\"\"\"\n",
        "\n",
        "# Matplotlib config\n",
        "plt.rc('image', cmap='gray_r')\n",
        "plt.rc('grid', linewidth=0)\n",
        "plt.rc('xtick', top=False, bottom=False, labelsize='large')\n",
        "plt.rc('ytick', left=False, right=False, labelsize='large')\n",
        "plt.rc('axes', facecolor='F8F8F8', titlesize=\"large\", edgecolor='white')\n",
        "plt.rc('text', color='a8151a')\n",
        "plt.rc('figure', facecolor='F0F0F0')# Matplotlib fonts\n",
        "MATPLOTLIB_FONT_DIR = os.path.join(os.path.dirname(plt.__file__), \"mpl-data/fonts/ttf\")\n",
        "\n",
        "# pull a batch from the datasets. This code is not very nice, it gets much better in eager mode (TODO)\n",
        "def dataset_to_numpy_util(training_dataset, validation_dataset, N):\n",
        "  \n",
        "  # get one batch from each: 10000 validation digits, N training digits\n",
        "  batch_train_ds = training_dataset.unbatch().batch(N)\n",
        "  \n",
        "  # eager execution: loop through datasets normally\n",
        "  if tf.executing_eagerly():\n",
        "    for validation_digits, validation_labels in validation_dataset:\n",
        "      validation_digits = validation_digits.numpy()\n",
        "      validation_labels = validation_labels.numpy()\n",
        "      break\n",
        "    for training_digits, training_labels in batch_train_ds:\n",
        "      training_digits = training_digits.numpy()\n",
        "      training_labels = training_labels.numpy()\n",
        "      break\n",
        "    \n",
        "  else:\n",
        "    v_images, v_labels = tf.compat.v1.data.make_one_shot_iterator(validation_dataset).get_next()\n",
        "    t_images, t_labels = tf.compat.v1.data.make_one_shot_iterator(batch_train_ds).get_next()\n",
        "    # Run once, get one batch. Session.run returns numpy results\n",
        "    with tf.Session() as ses:\n",
        "      (validation_digits, validation_labels,\n",
        "       training_digits, training_labels) = ses.run([v_images, v_labels, t_images, t_labels])\n",
        "  \n",
        "  # these were one-hot encoded in the dataset\n",
        "  validation_labels = np.argmax(validation_labels, axis=1)\n",
        "  training_labels = np.argmax(training_labels, axis=1)\n",
        "  \n",
        "  return (training_digits, training_labels,\n",
        "          validation_digits, validation_labels)\n",
        "\n",
        "# create digits from local fonts for testing\n",
        "def create_digits_from_local_fonts(n):\n",
        "  font_labels = []\n",
        "  img = PIL.Image.new('LA', (28*n, 28), color = (0,255)) # format 'LA': black in channel 0, alpha in channel 1\n",
        "  font1 = PIL.ImageFont.truetype(os.path.join(MATPLOTLIB_FONT_DIR, 'DejaVuSansMono-Oblique.ttf'), 25)\n",
        "  font2 = PIL.ImageFont.truetype(os.path.join(MATPLOTLIB_FONT_DIR, 'STIXGeneral.ttf'), 25)\n",
        "  d = PIL.ImageDraw.Draw(img)\n",
        "  for i in range(n):\n",
        "    font_labels.append(i%10)\n",
        "    d.text((7+i*28,0 if i\u003c10 else -4), str(i%10), fill=(255,255), font=font1 if i\u003c10 else font2)\n",
        "  font_digits = np.array(img.getdata(), np.float32)[:,0] / 255.0 # black in channel 0, alpha in channel 1 (discarded)\n",
        "  font_digits = np.reshape(np.stack(np.split(np.reshape(font_digits, [28, 28*n]), n, axis=1), axis=0), [n, 28*28])\n",
        "  return font_digits, font_labels\n",
        "\n",
        "# utility to display a row of digits with their predictions\n",
        "def display_digits(digits, predictions, labels, title, n):\n",
        "  plt.figure(figsize=(13,3))\n",
        "  digits = np.reshape(digits, [n, 28, 28])\n",
        "  digits = np.swapaxes(digits, 0, 1)\n",
        "  digits = np.reshape(digits, [28, 28*n])\n",
        "  plt.yticks([])\n",
        "  plt.xticks([28*x+14 for x in range(n)], predictions)\n",
        "  for i,t in enumerate(plt.gca().xaxis.get_ticklabels()):\n",
        "    if predictions[i] != labels[i]: t.set_color('red') # bad predictions in red\n",
        "  plt.imshow(digits)\n",
        "  plt.grid(None)\n",
        "  plt.title(title)\n",
        "  \n",
        "# utility to display multiple rows of digits, sorted by unrecognized/recognized status\n",
        "def display_top_unrecognized(digits, predictions, labels, n, lines):\n",
        "  idx = np.argsort(predictions==labels) # sort order: unrecognized first\n",
        "  for i in range(lines):\n",
        "    display_digits(digits[idx][i*n:(i+1)*n], predictions[idx][i*n:(i+1)*n], labels[idx][i*n:(i+1)*n],\n",
        "                   \"{} sample validation digits out of {} with bad predictions in red and sorted first\".format(n*lines, len(digits)) if i==0 else \"\", n)\n",
        "    \n",
        "# utility to display training and validation curves\n",
        "def display_training_curves(training, validation, title, subplot):\n",
        "  if subplot%10==1: # set up the subplots on the first call\n",
        "    plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n",
        "    plt.tight_layout()\n",
        "  ax = plt.subplot(subplot)\n",
        "  ax.grid(linewidth=1, color='white')\n",
        "  ax.plot(training)\n",
        "  ax.plot(validation)\n",
        "  ax.set_title('model '+ title)\n",
        "  ax.set_ylabel(title)\n",
        "  ax.set_xlabel('epoch')\n",
        "  ax.legend(['train', 'valid.'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VE6zp3q_HNTi"
      },
      "source": [
        "*(you can double-click on collapsed cells to view the non-essential code inside)*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Lzd6Qi464PsA"
      },
      "source": [
        "### Colab-only auth for this notebook and the TPU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "MPx0nvyUnvgT"
      },
      "outputs": [],
      "source": [
        "IS_COLAB_BACKEND = 'COLAB_GPU' in os.environ  # this is always set on Colab, the value is 0 or 1 depending on GPU presence\n",
        "if IS_COLAB_BACKEND:\n",
        "  from google.colab import auth\n",
        "  # Authenticates the Colab machine and also the TPU using your\n",
        "  # credentials so that they can access your private GCS buckets.\n",
        "  auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "R4jujVYWY9-6"
      },
      "source": [
        "### TPU or GPU detection"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 777
        },
        "colab_type": "code",
        "id": "Hd5zB1G7Y9-7",
        "outputId": "9a1d97ab-644c-4dd6-edc0-47e3b4e95829"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:TPU system grpc://10.41.182.106:8470 has already been initialized. Reinitializing the TPU can cause previously created variables on TPU to be lost.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:TPU system grpc://10.41.182.106:8470 has already been initialized. Reinitializing the TPU can cause previously created variables on TPU to be lost.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: grpc://10.41.182.106:8470\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: grpc://10.41.182.106:8470\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Running on TPU  ['10.41.182.106:8470']\n",
            "Number of accelerators:  8\n"
          ]
        }
      ],
      "source": [
        "# Detect hardware\n",
        "try:\n",
        "  tpu_resolver = tf.distribute.cluster_resolver.TPUClusterResolver() # TPU detection\n",
        "except ValueError:\n",
        "  tpu_resolver = None\n",
        "  gpus = tf.config.experimental.list_logical_devices(\"GPU\")\n",
        "\n",
        "# Select appropriate distribution strategy\n",
        "if tpu_resolver:\n",
        "  tf.config.experimental_connect_to_cluster(tpu_resolver)\n",
        "  tf.tpu.experimental.initialize_tpu_system(tpu_resolver)\n",
        "  strategy = tf.distribute.experimental.TPUStrategy(tpu_resolver)\n",
        "  print('Running on TPU ', tpu_resolver.cluster_spec().as_dict()['worker'])\n",
        "elif len(gpus) \u003e 1:\n",
        "  strategy = tf.distribute.MirroredStrategy([gpu.name for gpu in gpus])\n",
        "  print('Running on multiple GPUs ', [gpu.name for gpu in gpus])\n",
        "elif len(gpus) == 1:\n",
        "  strategy = tf.distribute.get_strategy() # default strategy that works on CPU and single GPU\n",
        "  print('Running on single GPU ', gpus[0].name)\n",
        "else:\n",
        "  strategy = tf.distribute.get_strategy() # default strategy that works on CPU and single GPU\n",
        "  print('Running on CPU')\n",
        "  \n",
        "print(\"Number of accelerators: \", strategy.num_replicas_in_sync)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "    \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Lvo0t7XVIkWZ"
      },
      "source": [
        "### Parameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "cCpkS9C_H7Tl"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 64 * strategy.num_replicas_in_sync # Gobal batch size.\n",
        "# The global batch size will be automatically sharded across all\n",
        "# replicas by the tf.data.Dataset API. A single TPU has 8 cores.\n",
        "# The best practice is to scale the batch size by the number of\n",
        "# replicas (cores). The learning rate should be increased as well.\n",
        "\n",
        "LEARNING_RATE = 0.01\n",
        "LEARNING_RATE_EXP_DECAY = 0.6 if strategy.num_replicas_in_sync == 1 else 0.7\n",
        "# Learning rate computed later as LEARNING_RATE * LEARNING_RATE_EXP_DECAY**epoch\n",
        "# 0.7 decay instead of 0.6 means a slower decay, i.e. a faster learnign rate.\n",
        "\n",
        "training_images_file   = 'gs://mnist-public/train-images-idx3-ubyte'\n",
        "training_labels_file   = 'gs://mnist-public/train-labels-idx1-ubyte'\n",
        "validation_images_file = 'gs://mnist-public/t10k-images-idx3-ubyte'\n",
        "validation_labels_file = 'gs://mnist-public/t10k-labels-idx1-ubyte'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Lz1Zknfk4qCx"
      },
      "source": [
        "### tf.data.Dataset: parse files and prepare training and validation datasets\n",
        "Please read the [best practices for building](https://www.tensorflow.org/guide/performance/datasets) input pipelines with tf.data.Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ZE8dgyPC1_6m"
      },
      "outputs": [],
      "source": [
        "def read_label(tf_bytestring):\n",
        "    label = tf.io.decode_raw(tf_bytestring, tf.uint8)\n",
        "    label = tf.reshape(label, [])\n",
        "    label = tf.one_hot(label, 10)\n",
        "    return label\n",
        "  \n",
        "def read_image(tf_bytestring):\n",
        "    image = tf.io.decode_raw(tf_bytestring, tf.uint8)\n",
        "    image = tf.cast(image, tf.float32)/255.0\n",
        "    image = tf.reshape(image, [28*28])\n",
        "    return image\n",
        "  \n",
        "def load_dataset(image_file, label_file):\n",
        "    imagedataset = tf.data.FixedLengthRecordDataset(image_file, 28*28, header_bytes=16)\n",
        "    imagedataset = imagedataset.map(read_image, num_parallel_calls=16)\n",
        "    labelsdataset = tf.data.FixedLengthRecordDataset(label_file, 1, header_bytes=8)\n",
        "    labelsdataset = labelsdataset.map(read_label, num_parallel_calls=16)\n",
        "    dataset = tf.data.Dataset.zip((imagedataset, labelsdataset))\n",
        "    return dataset \n",
        "  \n",
        "def get_training_dataset(image_file, label_file, batch_size):\n",
        "    dataset = load_dataset(image_file, label_file)\n",
        "    dataset = dataset.cache()  # this small dataset can be entirely cached in RAM\n",
        "    dataset = dataset.shuffle(5000, reshuffle_each_iteration=True)\n",
        "    dataset = dataset.repeat() # Mandatory for Keras for now\n",
        "    dataset = dataset.batch(batch_size, drop_remainder=True) # drop_remainder is important on TPU, batch size must be fixed\n",
        "    dataset = dataset.prefetch(-1)  # fetch next batches while training on the current one (-1: autotune prefetch buffer size)\n",
        "    return dataset\n",
        "  \n",
        "def get_validation_dataset(image_file, label_file):\n",
        "    dataset = load_dataset(image_file, label_file)\n",
        "    dataset = dataset.cache() # this small dataset can be entirely cached in RAM\n",
        "    dataset = dataset.batch(10000, drop_remainder=True) # 10000 items in eval dataset, all in one batch\n",
        "    dataset = dataset.repeat() # Mandatory for Keras for now\n",
        "    return dataset\n",
        "\n",
        "# instantiate the datasets\n",
        "training_dataset = get_training_dataset(training_images_file, training_labels_file, BATCH_SIZE)\n",
        "validation_dataset = get_validation_dataset(validation_images_file, validation_labels_file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_fXo6GuvL3EB"
      },
      "source": [
        "### Let's have a look at the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177
        },
        "colab_type": "code",
        "id": "yZ4tjPKvL2eh",
        "outputId": "d98e6355-a4a5-443c-8be3-dbd50d632a5f"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuMAAABQCAYAAACzvHtWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd1hVV9a433u5cOmiFKmiCCgqiL2iWLDEGo1JNNFUTWJM9XOcTMoXzWiimZjJJKapyRhLihobFgRFsWGhqChFpPdeL5fbzu8PfveMRrDARWfmO+/z+DwJZ9+99j5l77XXXmttWVVVlYCEhISEhISEhISExANH/rAbICEhISEhISEhIfF/FUkZl5CQkJCQkJCQkHhISMq4hISEhISEhISExENCUsYlJCQkJCQkJCQkHhKSMi4hISEhISEhISHxkJCUcQkJCQkJCQkJCYmHhKSMS0hI/Ftz9f0PSf/qG5OXbSvHQ8dTdvoMADe++Y6kv7x/T7+7n7IPknNPPUPubzvbrfzNXHxhEfm/77nn8tf/8RWXlv7pnspe/tNfSFv3Rava1ZbfSkhISLQWxcNugISExH8vx0PH02fVSpxGDG91Hb0/+rBdypqS7q+81Kqyqrx8YsaEMSH5MnLFv+9wfP0fX6HKzqHvZ2tNUt/ATd+bpB4JCQmJ/wYky7iEhMRDw6DTPewmSPybIb0TEhIS/9eQlHEJCYl24fL/LEddUEj8S68S2XcAGd9vQpWXz2G/XuTt2MXxUWO5MP85ABJee5Njw0KI6jeYc3PnU3v9+r/qucl1oPzceaJHjiFz048cGzKS6OGjyNv5e6vKaiqriFu0mMjgQZyZ9Thp674g9smnW+xP/p59HB89jqODhnHj629vufZHN4r83XvFsulffXOLS8vNZc/Pmw/A0QFDiOw7gMqEROqzszk3bwFR/QZzdPBwEt94u8U23e2+XfvwI+JefJnI4IGcnf0Equwc8XrZqTOcnDiFqH6DubbiryA0fxhzacxJMr79nqKDh4nsO4DT0x4Vr6nzC4h94ikigwdy4dkX0VRUiteqEi4R+/g8ovoP4fS0Ryk/d168drOLS96u3cQ+8RTJqz5pul//WN9if++l3wDaykouPPMCkcEDOTdvAQ35+eK1uhsZXHjmBY4OHErMhEcoPHioWRmaikriFr5CVP8hHB04lHNzn0YwGO7aNgkJCYn7RVLGJSQk2oWgv63B0t2N/t+tJ+xSHD6LXhCvVZy/wMjD4Qz8cQMAzqNCGBV5mLGxp7Dv3YvLb7fsH6wpK0NXW0foqWj6rP6Iayv+ira6+r7LXlvxEWbWVow9G0PQmtUU7N7bosy66+lc+98VBH26hjGnTqCpqqaxqLjlsh+upO9naxlz+gS6uloai0uaLTt4+xYAxsWdI+xSHB37BXP98y9xGjmccXGxhJ6Mxnv+Uy226273rfDAQbq/tphxF2Ox9u5C2udNCxVNRSUJS17H783XGXv+NNZdvKiKT2hRhs/Li3B9ZBJhl+IYsX+3eK1g/wECP1nF2NhTGLRaMjf9CIC6qJi4RS/TffFLjLt4lh7Ll5G45A005RXNyqi+dBlrL0/GnD1J98V3d/m5W78L9ofT/dVXGHfuDPYBPbm0dDkAOpWKi8++iNu0KYyJPUXw53/j2ocfUXc9/TYZmT/8iKVrZ8aeO8WYsyfxe/tNkMnu2jYJCQmJ+0VSxiUkJB44vq+9isLaGjNLSwA858xGYWuDXGmB7+uvUpuSira2ttnfyhQKui95Bbm5Oc6ho1FYW1OfkXVfZQW9nuKISHxfX4KZlRW2fr64PzqjxfYWHT6C85hQOg0eiFxpgd+br4G8ecWsqewYOg4cgNzCAr83XoP70OHk5goaCgppLC7BTKmk48ABLZa9233rHDYeh75ByBUK3KdPpTY5BYDSEzHY+vriOnkicnNzvJ9dgIWT07030ih/9qPYdOuKmaUlbo9MEusv2Lsf59GjcA4djUwux2nkcOz79KH0REyz9ShdnPFe8DRyhUJ8J+4o9y79dg4d/a9n9fYbVCUk0lBYSGn0Caw83fF8bBZyhQL73r3oPCGMosMRt8mQK8xpLC2loaAAubk5nQYNRCYp4xISEu3Av2/EkISExH8tlm6u4n8Lej1p676g6FAEmooKZPImG4G2ohJzO7vbfmvu4HBLsKPcyhKdqr5ZOS2V1VRUIOh0WN3Ujpvb9EcaS0qwdP3XdYW1NRYODi2WvbleMyurFss2R48/LeX637/k7GNPYG5vT9fnn8Vzzuzbyt3LfbtZwZZbWaGrV/2rPze1USaT3bH/LWHhfFP9lpboVE31NxQUUHQogpJjx//VXp0Ox6GDm63H0s3tnmXeS79veVY2Nph36EBjcQkN+QVUXbpMVP8hN9Wnw33G9NvkdHvxedK//IqLzy4EwOvJOfi8tPCe2ykhISFxr0jKuISERLvRoiXxpr8X7D9ASdQxBm3ehJWnB7raWo4OGIpA8z7MpsCiUydkCgXqomJsunUFQF1Y1GJ5pYszdTcyxP/XNzSgqapqsezNlnq9Wt1i2eZuj9LZmT6rVgJQeTGOC8+8QMfBA7Hx9r6lXFvum9LZ+Zb+CoJwx/7fr0XY0s0V95nTxX7clfuo/176rS76V1909fVoq6tRdnbB0s2VToMGMWjzprvKUdja0POd5fR8Zzm1ade5MP85OgT2wXH4sHtuq4SEhMS9ILmpSEhItBsWjo405ObdsYy+vh65hTkWDg7oGxpI++zv7d4umZkZnSeMJ/0f69E3NFB3I4OCPS37jHeeNIHS6ONUXozDoNFw/YsvwdC80tt50gRKoqOpjE/AoNE0BSS2oB9bdOoEcvkt96jo0GFRMVZ0sAeZDJns9qG6LffNOXQ0denpFEVEYtDpyN68FU1ZWYvlLRwdacgvuOcARvcZ0yg5Fk3pyVMIej36xkbKz52/o8J/r9xLv0uPx/zrWf39SxyC+2Ll5obLmFDqs7LI37MPg1aLQaul+vIV6tJv3FZHybHj1GdnIwgCCjtbZGZykEtTpoSEhOmRRhYJCYl2w+flhdz4+lui+g8hc+MPzZZxnzkdKw93okNCOTV5Gg7BfR9I23p98B662lqODRvF5WV/xm3qFOQW5s2WtfPzo9f/vs+lt5cRPWI05vb2KF07t1g24P13ufTmUqJHjMbM2hoLx07ILSxuK2tmZUX3V14i9omniOo/hKqES1RfTuLsnCeJ7DuA+JeWEPDeO1h38brtt225bxadOhL8j89J+9s6jg0ajio7G4f+/Vos7zp5EgBHBw3nzIzbXWb+iJWbG/2/+YqMb77n2JARHA8ZS9aGHxCEtmcjuZd+u0+bQvqXX3N00DBqkq4S9Lc1QJO1e+CPGykKP0j0iFCih48i9dN1GDSa2+pQZWdz4ZkXiOo7kNg58/CaNxfHoUNuKychISHRVmRVVVXttxcsISEh8R9C6trPaCwrI2jtxyatV1dfz9EBQwmJPIS1l6dJ65aQkJCQ+M9HsoxLSEj8n6TuRga1KakIgkDVpcvk7dxF57BxJqm75Gg0+oYGdCoVqZ98iq2/H1aeHiapW0JCQkLivwspgFNCQuL/JPr6ei699T+oS0pROjnS9flncRlvKmX8GJeX/RkEAfvA3vT9+2dSWjwJCQkJiWaR3FQkJCQkJCQkJCQkHhKSm4qEhISEhISEhITEQ0JSxiUkJCQkJCQkJCQeEnf0Ge/QocODaoeEhISEhISEhITEfy0Gg4Ha2trb/i5ZxiUkJCQkJCQk/gvYsWMHYWFhLF++nOTk5IfdHIk/YGjh4DQpm4rEfxyJiYnEx8dz8eJFrl+/jpOTE7Nnz2bmzJkoFP/dr3R8fDwHDx7k1KlTDBo0iD59+tCvXz+6du2KRTOHykhISPyL+vp6rl69ypEjR0hISKCqqgoXFxfeffdd+vTp87CbJyHRJlQqFXv37uXixYvk5OTg6+tLQEDAw26WxD3w0DQXnU5Hbm4u4eHhODs78+STTz6sprQbly5dYv369bz11lv4+/tjZmb2sJv0H8+GDRuIiori+vXr5OfnU1paio2NDcXFxZiZmREaGkrHjh3bvR1arZZdu3YxdOhQunbt2m4ycnNzOX36NA4ODmRkZHD69GkSEhLIysoiNTWVzp07M2zYMJ544gmGDh1qUvmJiYl8//33DBgwgEcffZROnTqZtH4Jibq6OhISEjhz5gzW1tYsWrQIpVLZLrLy8vI4fPgw+/btIzk5mcLCQhoaGrC1tUWlUrF48WLGjx8vjdMSbUIQBE6cOMGBAwfw9/fnqaeewtra+oHIzsjI4Nq1a9TV1QFQXl7+QOQ+DDQaDaWlpWRmZlJcXMylS5eora3lhRdeoHfv3v9xqWTbVRnX6XRoNBoE4V/ZE83MzFAoFFRUVHDo0CF++OEHnnnmmfZsxkNBq9Vy6NAhNm7cSGBgIN7e3g/sg/xvpaioiB07dnDmzBn0ej0dO3ake/fuVFRUEBsbi6WlJW5ubiZRSrOzs+ncuTOWlpa3XRMEgerqajZs2IBSqWwXZby+vp4rV67w66+/Ehsbi729PTk5OeTm5qJSqcQ2ZmdnU19fT79+/UyujO/atYsffviBuro6wsLC2l0Z37NnD5mZmbi5uTF69Gjc3NzaVZ7Ew8X43e7cuZOEhARGjBhxy1xhKrRaLQkJCURFRREeHk5cXBwymQxXV1fc3d3JzMwkPDwcQRDw9/fH29sbufzBeHCq1WoiIiIoLi5m5MiR9OrVy6T119bW8vPPP1NTU4ONjQ1z5szBycnJpDKMxMTEcOHCBXx8fAgLC8PW1vauvyksLCQmJobJkydjZ2dnEgWqsrKSrKws8vPzqayspKGhAXd3d9RqNX5+fnTp0gUHB4d2U9Zqa2v5/vvvOXjwID4+PsyYMeOBzP0ajYbt27eTn5+PXq/H1dW13Z71w6Kuro7r16+TmZlJaWkp2dnZZGRkUFJSQnJyMnV1ddTV1bF+/XqT7RRrNBqKi4tJTk4mMzNT9Pf29vZmzpw5JpEB7aiMNzQ0cOHCBfLy8tBqteLfbW1tcXBwIDMzk71799KxY0dGjRplcvlGhSknJwe1Wo2Li0u7WTD/iMFgIDc3l3379gGQlZWFXq9/ILLbE4PBQFJSEmVlZeh0OlxcXLCysqJz587Y2dm1u0UpJSWF3Nxc9Ho9gwYNYtiwYVhbWxMfH8++ffuIiYkhMTHRJEppREQEEyZMoEuXLrdNzAaDgZKSEi5cuEBmZmabZf0RnU5Heno6mzZtYsuWLej1enx9ffH398ff3x9o2o68fPkyFRUVlJWVkZWVRXl5OY6OjiZrx/nz59Hr9Q/MwrB582YOHz7M4MGD6dat2wNRxq9duya+U+bm5kCT8qbT6bCwsCA4OBhnZ2fJWmpCiouLKSgoID4+ngMHDnD69GlsbGzw8PBoFzezhIQENm7cSEREBCUlJXTq1ImAgAAGDhxIUFAQ0dHRbN26lUOHDrFo0SI8PDxaPZHX1dXR2NiItbU1VlZWdy0fHx/PunXrSElJYfny5bi7u+Pg4NAq2X9EpVKxf/9+3n//fUpKSnBwcMDPz4+RI0c2a2RoC4IgsGHDBrZv386AAQPo168fNjY2dxw7VCoVO3bsYOPGjVhYWDBu3DiTKOTp6en8/PPPxMfHk5+fT01NDQEBAVRXVzNs2DD69+9PcHAwfn5+2NnZmXThpdfriYuLY//+/dTV1XHlypVb9J/2QqvVcurUKbZt20ZZWRkdOnQgJCSEvn37trvs9katVpOamkpBQQFFRUWcPXuWhIQEiouLqayspL6+HgB7e3sUCgVXrlxp0S/7fmlsbOTGjRscPXqU8PBwEhISKC0tRSaT0bdvXwICAkxmhTf5yKfX66mtreXChQusWrWK8+fPIwgCSqUSjUaDnZ0d7u7uFBUV0aVLF/72t7/Rv3//+5Kh0+kwMzNr8QZotVrKy8s5c+YMv/76K6WlpYwZM4alS5c+kBWqVqvl2LFjxMbGAjBkyBBxov9PRa/Xk52dLT7T+vp6RowYgaurK1OmTKFr166YmZlhbW2Ns7Nzu9xntVpNly5dcHFx4e2332b69OnU1dVx4MABIiIi0Ov1JCUlmUTWgQMH6Nq1Ky4uLrf1Ra/XU1JSglarbRclTa1WEx8fz7Zt2zAYDNjY2DB37lzeeecdsUx+fj5vvvkm0dHRlJSUEB8fT3JyMiNHjjRZO4wLEVMNbHdDJpMhk8mws7O77XsxGAyo1WqUSqVJ7/nGjRs5dOgQFhYWuLi4AFBdXU11dTUAr732GuPHj8ff37/drKWCIKBWq8VFgFarRaPRUF1dLS5yLS0tsbOzM5kLhyAINDQ0AIg7LdA0thrbYsTMzAx7e3ugacJrjaJqMBhoaGigrKyMAwcOEBUVxalTpygtLQXAycmJ5ORkUlJS6N69+z0psvfKTz/9xP79+1Gr1XTv3p2QkBAWLFhAr169sLOzIzg4mN27d1NVVUVmZqa4EGsNV69e5fr16/j7+xMUFHRHpbeyspKvvvqKxMREamtrSUpKIjMzk379+rW2q+h0OioqKigvLycnJ4e3336bkpISAKqqqggPD0cmk9G7d28cHR1NNi9VV1dz/PhxoGnxU1dXhyAILc7RgiBQVlbGRx99REVFBWvWrMHPz4+ePXu2eUFWWFjImTNnuHbtGnK5HDMzM+Lj4wFITk5m27ZthIaG8uyzz5rcrbG2tpYff/yR+vp6nJycqK6ufiDjZ1lZGStXrqSkpARBEAgKCuKxxx5j4MCB7S67vUlMTGTdunVERkZSVVV1yzWZTIaNjQ1OTk4EBQVhbW3Niy++aDKreE5ODrt27WLr1q1kZ2eLC1q1Ws3ly5fZuHEjq1atQhAErK2t2zRHmFwZLy0tZefOnbz33ns0NjaiVCrp06cPHh4eXL9+ncTERGpqaujTpw+rV68mJCTkvuo3rlTc3Nzo0KFDs53Pzs5m27ZtbN68mczMTORyOTdu3CA0NPS+5d0vgiBQU1NDeHg40DSRDR8+vN38IO8XvV6PRqMBuK8Jr7q6mg8//JDffvsNCwsLrK2tOXz4MBqNhuzsbCorK0lPTyckJIRly5YxbNgwk7d90qRJBAUFIZfLxW1GW1tbfH196datG2lpaZw8eRKDwdBmxUmlUpGTk0NdXV2zynh5eTk6na5d0n/K5XLReldRUYG3tzdubm63TJxdu3YlMDCQuLg4cZFrY2Nj0naMHDmSrVu3UldX1+47OwaDQZy0Ro0adZtVvKKigvPnzzNo0CCcnZ1NIlMQBKqqqhg/fjxz585l0KBBQJPSUlBQwKFDh1izZg0xMTF8//33JrNYGmUb+6xSqYiPjyc3N5eysjIKCwu5ceMGR44cISQkBAcHB3r16sWUKVPu23DRErW1tZw5cwaAixcvis+3tLSUa9euiUoyNO1mTpo0CblczsSJE+ndu/d9v2s1NTWcOHGCb775hhMnTiAIAhYWFuI9bWho4LfffuPChQusX7+esWPHmqSfxva7uLjQt29f5s2bx/Dhw8XFBUCPHj1wd3enqqqK48eP8/TTT7famLB37162bdvGiBEjeOONNxg8eHCzCqler2fbtm0cPnyYmpoabG1t6dKlS5vebZ1OR2ZmJhs3buSnn36iqKhIvGZhYYHBYOCLL77giy++YMmSJbz++uv4+fm1Wt7NfTl69KioKBktlHdCEARUKhUVFRVA08Lkjy6trSUoKIgFCxYQFRWFpaUlzs7O5ObmAnDjxg2Sk5PZv3+/aPAw1c68VqslNTWVHTt2YG1tzYIFC/jnP/+JSqUyyZzUEgaDgStXrnDx4kVxV2bgwIF4e3v/x/lN/5G6ujo++OADTp06JRpjzM3NkcvlyOVy7O3t6d27Ny+99BITJkww6SK+sbGRiIgI0fWnR48ePPXUU4wdO5aUlBQWL14sLr4zMzMZNWoUnTp1av09r6qqElr6d7/odDph7dq1glwuF8zNzYVHH31U+OWXX4Ti4mJh//79QlhYmGBpaSkMGzZMOHr06H3Xr1arhZMnTwqurq7CkiVLhPT0dMFgMNxW7vDhw8KkSZMEQAAEmUwmdO7cWdi2bdt9y7xfamtrhQ0bNohyfX19BZVK1e5yW8JgMAh6vV7Q6/VCTU2NcO7cOeGjjz4SPvjgg/uqJz8/X+jSpYsgk8mEadOmCRcvXhT+9Kc/Cf7+/oKFhYV4r6dNmyacPHmynXrTPFlZWcKiRYsEuVwu9OnTR8jOzm5znS+//LLw1ltvCVeuXLntmkqlEnbu3CkoFArhu+++a7Oslqiurm7xXhoMBuHdd98VPD09hW7dugnffvutyeX/85//FCwtLQVzc3Ph4sWLJq//ZjIzM4XBgwcLSqVSOHz4sNDY2Che02g0QkREhODj4yOkp6ebTGZlZaUwZMgQYcmSJcLVq1dvu67VaoXo6Gjh5ZdfFrKyspoda1qDSqUSioqKhKtXrwqHDh0SPv74YyE4OFiws7MTfH19BZlMJshkMkEul4vflVwuF5YvXy6o1eo2y29oaBDWrFkjypDL5YJCoRDMzc0FCwuLZv8Zr9va2gqvvPKKoNFo7kvm+fPnBXNzc0EmkwlKpVKYM2eO8NNPPwlxcXFCbGys8Mknnwg9e/YU5HK50K1bN6G+vr7N/bxXqqqqBF9fXwEQZs2aJVRUVLS6rrCwMPFeLVy4UCgtLW22XFxcnNChQwfx+Y4cOVI4cuRIq+UKgiCcPHlSCAwMvOWdMT7bWbNmCaGhoYKdnZ0ACA4ODsLHH3/cJnlGNBqNsHr1alHeypUrherq6jv+Rq1WC+vXrxfbt2zZMqG2ttYk7WkJg8EgXL58WejWrZugUCgEuVwuREREmKx+tVotREZGit/L6dOnhXXr1gnnzp1rVx0gMzNT8Pf3F+RyuSCTyYSXX35ZSE1NbTd5N6PX6002LjbHggULBAsLC0Emkwk2NjZCWFiYsGLFCmHDhg1CVFRUu8kVBEG4dOmSMGfOHMHJyUmYO3eucP78efGaTqcTJk2aJPTp00eYNGmS4ODgIPzyyy93fe8FoWleaU7fNpllXKfT8fPPP/PRRx8hCAJbtmyhf//+eHp6sm3bNn766SeSk5MJDQ3l3XffbdV2ul6vp6qqiqKiIr766iv27NnDwYMH6d27t7jqTE1NZfPmzRw7dkz8nY2NDRMnTmTevHmm6m6zqFQqTp48yZIlS4Cm1f+sWbMeSro9jUZDeXk5ly5dIi0tjRs3bnD27FkuXLiAhYVFm9wZhg4dSv/+/dFoNGzbtk20tEPTFmFmZqZJ3SXuhtGFAZp2Itzd3dtcp7W19V3dIQwGgxi13h7Y29u3eB9/+OEHtm3bRkFBAd7e3u0i3+iLp9PpSE1Nxc/P7xaLoikJDw/nypUr2NjYoFQqb7MuCIKAVqultLSU7t27m0RmRkYG9fX1omX0jygUCkJDQwkNDTWJPIDLly9z5MgRfv/9dy5evIilpSU6nU50GamtrWXUqFF4eHjQtWtXfvzxR4qKijAYDKSkpBAfH9+mXae6ujo2bdrEhx9+iLe3N/7+/gwZMoRu3bphY2NDSEjIbbsSxvif7OxsALp163bfY1rHjh0ZMGAAXl5ezJ8/n9DQUOzs7MTrPXr0wM3NjUWLFlFaWsrevXuZO3duq/t5rxjfK1PERqSmplJbW4vBYMDR0REXF5cWd0TXrFkjuge5urryl7/8hTFjxrRJvlqtpqqqCh8fH4YMGcLMmTPx8vK65X3585//zObNmykqKhJdsUyJlZXVPb0fWq2WgwcPAjBt2jTef//9ewr4bC0NDQ1s2rSJ9evXk52djSAIzJw50yRzhZHq6mrWrl2LVqslLCyMoUOHMnz4cJPV3xxGXSs9PR1BEOjbty/PPPPMA4mP02q1rFu3DmtrayZOnCjGM6nVavLz83FwcMDR0ZHq6upW7SCXlpZy5swZdDodvr6+LFmyhNmzZ+Ph4WHqrtwRLy8vJk6cKO6cQtPu9bx583jmmWdISkrC3t4eZ2fnNnlAmERLNBgMFBQUsHbtWurq6li6dKkYHZ2SksL27dvJzc1l7ty5PPXUUwwePLhVciwtLenTpw/BwcEkJiaSl5fHtGnT2L17N4GBgSgUCq5du0ZOTo6oIMrlclxdXXn55ZdN0dU7YnTR0Wg0yGQyunTpwqJFi0yujOv1esrKyoiLiyMmJkbc2lSpVMTExIh+gzk5OahUKszMzOjQoQPdunXj3XffZeLEifj4+LRKtkwmY8qUKRw4cIA1a9ZQVlZ2y/X4+Hg+++wz1Go1CxcubHNf7wWdTkddXR0Gg4H09HST3G8PDw8qKytN0Lr2Ydu2bRQVFSEIAmPGjGkXt6CdO3ei0+kQBIH09HRqamraRRkPDw/n3XffRa1W8/LLL+Pn59esL2tFRQVr167l999/N4nc3NxcGhsbsbKyeiA52pcvX84vv/xCYWEhOp0Oa2tr3N3dcXd3Z+TIkfTu3ZupU6eiUCgwMzOjtLSUgoIC9u3bR1VVFX369GHIkCGtlq9SqYiNjWXjxo3Y2try5ZdfMmHChFvib5rbSreysiIkJERcGBp9+++H7t27c+zYMbFvf5Qjl8tRKBTIZDKsra2ZPXt2K3t5fxgMBpKTk8nPzwdok4vKwoULSUhIwGAw8PTTT/PKK6/csuAwEh8fz6FDh9BqtdjZ2fHOO+/Qv3//No9bY8eOJT09HUDcxr/5OR06dIhdu3ZRVFTEmDFjGDFiRJvkQdNclJWVxfbt2wHw8/Nj+vTpd7yHDQ0NREVFiX7cxgW4KTEYDNTX15ORkcHOnTvZuXMnGRkZ6HQ6rKysCA4OZuXKlSbLXqNSqcT52MzMjDVr1rR7Rh6dTse2bdv4+uuvRfeeb775hv79+7f7eKbRaFi4cKEYqLps2TJkMpkYJ2Bsz81/gyYj1+TJk3nvvffo2bNni/WXlJQQFhZGRkYG7u7uLF26lNmzZ981O0xZWZnJMsgEBAQQHBxMWVnZLXE1Rm72X3/vvffo27fvw1fGq6urWbFiBWlpaYwZM4bly5djZ2fHmjVr2LVrF+np6XL/4dUAACAASURBVCxYsICFCxfSo0ePVlsg5HI5HTp0YMiQISQmJgJNE2p0dDS+vr7Y2dmRmJhITk6O+Bt7e3tCQkLaPZAhPz+fX3/9lZ9//hmZTIaTkxPbt2+na9euJvXbamxsJCYmhrVr11JTUyMGqUDTKtnodxcQEMD06dMJCgqid+/edOvWDUdHR5RKJZaWlq0OgjMYDMyfP5/S0lIqKipusYobr2dkZBAdHf1AlHG9Xk9+fj4HDx5EqVTy+OOPm6Tee3lmcrmcbt26mUTe/XDx4kUKCwvFLCvDhw9vF0tIUFAQp0+fFi3Erq6uJpchCAJnzpxBpVJhaWmJr68vDQ0NREZGsnXrVjEA3Oh3qdPpTCq7U6dOdOjQ4YEEWGs0GuRyOQMHDmTixIkMHz4cX19f7O3tUSqVKBSKWwL+jhw5QkxMDFVVVXh7e+Pp6dnqCV6r1XLlyhXefPNNsrKyePvtt+nZsyfV1dU4OTnd9X1vjQL+x9+35M+p1+vJycnh7NmzaDQalErlAzvASqvVEhUVhU6nw9LSks6dO7f6Hhv9nqFJwWzO0mswGPjqq6/EHbUlS5Ywa9Ysk8RByOXyFu/boUOHWLVqFXl5eQCEhIS0aWFnpLa2lrVr15KWlgbAsmXL7uq3W1ZWxvvvv3+bIccUqNVq0tPTiYqKYvfu3aSkpKBWq2loaECn0zFhwgSefvppxo4di4uLi8kU5oKCAr788ksEQWDq1Kl4eXmZpN6WEP5/trjvvvuOkpISrK2tef311/Hx8Wn3sUyr1fLOO+9w8ODBFgNU5XI5VlZW2NraIggCXbt2pX///kyZMoV+/frRuXPnFvtVWFjIc889R0pKCoIgsHbtWiZNmnRbzE5FRQVXrlwhNzcXnU7HkSNHxEMAP/30UwICAtoU7G9ubk7v3r05fvw427dvx8vLi6lTp6LX60UdTBAEXn31VR5//PE2xxS1WRnX6/WUlpaSkJCATqejU6dOdOzYkQMHDrB9+3YyMzOZP38+8+fPx9/fv80vir29PW+99RYlJSXs3r0bg8HAhg0bGDx4MAMGDECj0dySCcDBwYHBgwe36wtaUVFBZGQkGzdupLGxEU9PT/7617/Sv39/k2Z+KCgo4MCBA+zcuROFQsGaNWvQaDSkpaVRUlJCcHAw9vb2yGQyXFxcsLe3x8bGBmtra5NmoUhNTUWr1SIIAoGBgcyYMYPGxkYOHTpEUlISOp2OxsZGk8i6G9XV1aSnp6NWq7Gzs+Oxxx5rd5mNjY1cunQJmUxm8rzAdyMxMZHXX39d3GodMWIEwcHB7bLF27NnT+RyOYIgYGNj0y7uVjqdjqqqKgwGA4IgEBUVRUxMDGlpaWRmZt7iBqRQKNBoNDQ0NJgkUCchIQEfHx/c3d0fiCvZrFmzGDduHO7u7nh4eGBnZ4elpWWzCkFCQgLfffcdhYWFKJVKZs2axfjx41stOyMjgx9//JH09HQ0Gg2bN29m3759yGQyhg0bhoODA/369aNfv354enqaPPVdS5SVlREZGcnPP//MhQsX8PT05MMPP3wgsjUaDRcvXuTnn38GmgKHAwICWj1XGINgHRwccHBwaPadEgSBvLw80XLo7u6OjY1Nu1lRi4uL+eWXX/j222/JysrC2tqaJUuWMGfOnDafG6BWq0lISCA8PFxcJN8tNWFjYyOZmZmUlJSI9+uNN95o1T0PDw8nKysLg8HAjRs3yMzMpLq6mpqaGsrKyigvL0er1dK5c2fmzZtHaGioqAga50lTUVNTw5kzZzAzM+Pxxx+nuLhYdNXw8PAwedatyspKVq1axeXLlxEEge7du7Nw4UIcHR3bNWhTq9Xy8ccfs337dtzd3Zk3b56ohMpkMjp27Chm4bK3txffMaVSia2tLR07dsTKyqrF991oXDtz5ozoYfD7779z7Ngx0tPT6datG1VVVeTn56PValGpVDQ2NoqLk4aGBszNzXnyyScZP348y5cvb1OKXEdHR+zs7Dh79iwrVqzg2rVr9OvXjz/96U8UFRUxduxYXnvtNdzd3dv8jNs8A8nlcpRKJUqlEkEQKCoqYu/evXzzzTekp6fz1FNPMX/+fHr16mUSa4eZmRk+Pj4sWLCA48ePU1lZSUZGBl988QXPPPMM165du2X7QC6Xt6svGjTlvz548KA42I0bN45HHnnEZBOaIAikpqby22+/cfjwYezt7XnuuecYOnQoMpmMoKAgLl++zLBhw8RtEnNzc5N+lPb29ixbtoxdu3YRGxuLIAjY2dnx/PPPExoays6dO0UfYzc3t1a7It0PWq2Wq1ev8vvvv2NpacnAgQMJDg5ud7nC/09Fp9fr2b9/P9bW1nh6erZ75LpOp+OXX37h8uXLqNVqzM3NGTVqFH5+fu0ymd+s8LZH34zjRWRkJIIgoNPpOH36NDqdjvr6+tt2XaBpIXj+/HlGjx7dZvmxsbF06dKlWQVC+P9ZkZKTk8XUcMbMJq29F8HBwchkMiwtLe+o/BcVFbFp0yaSk5NRq9UMHTqU0aNH06VLl1bJ1el05OTkEBMTIy6SS0tL6dixI4IgiDl7IyIicHFxYcKECUyZMgVfX992TecYFxfHnj17iIyMFC1aixcvZurUqe0i84+o1WrOnTsnLmzfe+89k2RHMh620pyCWVdXd8v8VFxcTFJSEj4+Pjg5OZnMXUOn03H48GFxvE5NTcXb25tly5YRFhaGl5dXm5UHvV5PdXW1aOE2noNgfGcqKipoaGhAr9ejUqlIS0vj7NmzpKWlUVNTg4WFBdOmTSMgIOC+vymtVsv27du5cuUKOp1OVMK1Wi0Gg0FU9JVKJb6+vkybNo0RI0a0S/YrvV5PfX09tbW1mJmZsWPHDrZs2SK6Ia1YsYKePXuazCDY0NBAYmIiO3fuFN0nZs6ciYuLC5GRkWJaSQAXFxf8/f1xdXU1yRi+e/dutmzZQnFxsehm0rFjRwIDA+nSpQu2trbieGpubn7fOp8gCDQ2Noq6BMCJEyeQy+VUVlaSlJSERqMRD94xusCYm5szdepUbG1t2bdvn6gHjhgxggkTJrT6uQcEBDBz5kyKi4uJi4sTdxKvXbvGwIEDWbFiBd27dzeJMafNNchkMhwdHVm4cCHFxcVcv36dr776itjYWPr27cvcuXPp27evSfNOm5ubM2LECGbOnMnmzZvRaDRERUWJpxYag/keBBkZGRw5coSzZ88ik8nEhYIpD1+5ceMG27ZtY+fOnQCMHj0aT09PUlJS6NGjB66urtTV1TXrn2gqrK2tefzxx+nSpQvHjx9HpVLh4ODAtGnTqKur4+rVq6LS0qlTJ5OkzLobVVVVXLp0idjYWGxtbXn00UdxdnZGpVKRm5uLvb19q7edjYOZkdraWm7cuEFSUhJZWVmcOXMGg8HAli1byMjIYNasWYwaNarNC86srCzy8vKaTSVoDGwzBvt169atXYMq2xNBEKioqGDr1q3cuHED+Ncum5eXFyNGjKBr167k5eURGxtLUVGRmFLS6N/bVoqLi8X8+EbUajW5ubmcO3eOtLQ0MfVadnY2VVVVzJkzp9VW+Xv5PgVBYOfOnRw4cICamhqCgoJ46qmn2uQHapy8BEEgODiYwYMH4+bmhq+vL9AUqJuamsrFixc5e/YsZWVl2Nra4uzsbLIczHV1deK3U11djVar5cKFC5w8eZKKigoCAgJ44okneOyxx0yWuvJOGN81Y4xNz549GTFihEkUlqqqKk6cOIFarb7l3TK+8ze7UR45coTU1FRxh8ba2hp7e3t69OjRqgNbtFotWVlZopvGuXPnqKmpAZre7aSkJJRKpag4t+QucC80NDSQkpIiuil4eXmJaTmN6faKi4vF3az8/HxSUlKora1Fp9Ph5OTEq6++2qrvyZje1mgNVSgUODg4oFKpUKlU4vhpMBgoKiriyJEj7eaqWlNTw/Xr18VUpUZdJCAggFOnTvHII4/QpUsXky0EKisr2bdvn+hyFBQUxOjRo9m9eze//fYbNTU14vxlPFTxiSeeaHPgo9EtJiMjA0EQyMnJoaqqiqSkJFJSUsRg97YYP43P1dfXl+vXrwNN45fx/BJoUpCDgoJueXcVCgXDhg3DysqKvn37ijtB+/btIzAwsNX33tHRkbFjx4qxBqdOneL69etYWloyZcoUk40ZYCKfcVtbW+bOnUt4eDi7d++muLgYaDouVKvVikFSprSuOTs78+KLLxIfH8+VK1eoqqri0KFDt5UzHpjRHtTV1XH8+HEOHDhAXl4ezs7OjBs3zuSZRDIyMoiKihKV75KSEnbt2oVKpWLEiBEMGDCg1Vaze0Uul+Pi4sL06dNFdyBLS0s6derEDz/8wLVr16ivr8fc3BxHR8d295nTaDQkJSURHR1NdXW1mKP38OHDlJSUkJqaSmBgIHPmzGnV81er1RQXF3Pu3Dlu3LhBQUEBly9fJjY2lqysLNG6VVhYSEJCAoGBgQwfPrzVCtPly5fJz8/nwoULpKSkiLncKyoqxMmuuLiY1NRUoGmQnTFjxgOPLDcVOp2O5ORkfvzxR/FvFhYWuLu78+ijjzJ58mR69epFWloaDg4OhIeHm9zH1N3dHa1WS319PSUlJZSUlJCVlUVcXJx47LG/vz/Ozs6cPn2aAwcOMG3aNJPmsjVijLXIycnhhx9+ICcnBxcXF+bNm9fm5yyTyfD09OTJJ5/E1dWVsLAwXFxcxElTp9NRXl7OsWPH2L9/PxcuXGDfvn24ubkxceJEk1j0SkpK+Pbbb7l69SplZWWiMqxWq/Hw8GDUqFE8++yzJs3lbsRoOa2oqECtVmMwGKiqqhKzSxljkU6ePImXlxeurq5YWlre93xlZWWFmZmZuDt88uTJ2+pQqVTi/AhNp9xevHgRLy8vHBwcUCqVeHt7M3369FYp48YF7pYtW8TTga2srMS4gE2bNnH06FGCg4MZP348YWFhrYo30el0FBQUcPz4cdE6WV5ezubNmyksLBSz/5SUlIi7MTffC3Nzc7p3797qHS6jO0inTp3EsyDMzMyoqqqiuLhYdG9raGggLi6OTZs2MWXKFMLCwlol707k5+dz4sQJoGme9PPzw8nJidDQUFasWEFWVpbJ3DaN56wYdR2ZTMYTTzxBZWUlX3/9NfHx8WIQdENDgziPBQQEtHmuMJ6rMWjQoNtOLr18+TLl5eVoNBpmzJjRauVXLpfj7u7OCy+8IN5TT09PnJycxIVtnz59GD58eIs6xpAhQ0hKSqKgoICEhIQ2J2Lw8PBg2rRpCIIg5nKHpkVRTk4OXbp0+fc5gVMQBOrr67GxscHBwYFOnTqhUqlISkriu+++o66ujtGjR5ssytXI8OHDWbx4MZ9++il5eXm3WcTNzc3p2LFjuymG169fJzIykqSkJCwtLenRowdz5841uf9phw4d6NGjh6gAJiYmiq4Shw8fZtq0aXz00UcmlXknbv6o8/LyOHz4MIWFhUCTVTwwMPCOkdL3glEBra6uRqPRYGVlhbm5OfX19dTU1FBQUMCOHTs4cuQIZmZmCILA77//zoULF0TlXKFQtCrgT6PRkJOTw/nz5zlz5gzl5eWUl5eLJ8kaUxiVlJTwyCOP8Nhjj9G1a9dWKeJGC8O3335LdHQ05eXlWFlZ4e/vj4eHB6dPn27WEuzj48P8+fPbLa1he6PVaklOThazPwAEBgYyZcoUXnjhBdHtx9HREYVCQV1dnbgzZCrGjx/P6dOnOXv2LHFxcSQnJ3P9+nW0Wi3z5s1j4MCBBAYGApCUlCRmrzE1Rn/HH374gejoaBISErC0tGT8+PHMmzevzeOXlZUVgwcPbtF1TKFQ0LlzZx5//HECAwNZuXIlERERODo6MmzYMJPs8tXW1rJ3714aGxvFuKKOHTtSXFyMQqEQT0o0FUbXiLKyMoqKikhJSSE1NZWKigq0Wi0FBQWcPn1a3GVKTEzkww8/JCQkhEGDBhEQEICnp+d9uY0MHDiQkpISiouLqaioEHdV7oRSqcTNzY2+fftiYWGBQqEQXVZaQ0lJCRs2bKCiokJ0wfLx8RGP7L548SJ5eXlERESIu8ivvPLKfc9ZZWVlHDt2jLi4OADR3SkhIUEsY2ZmJrqwajQaMbmAmZkZHTt2ZMKECa3qo5HQ0FBGjhwpGobkcjl1dXVUVlbS0NCAIAjk5+fz6quvkp6ezunTpxk/frxJjYIGg4G8vDyio6ORyWQ4ODiwdOlSAgMD6dixI6tWrcLCwsJkMqurq7l06RIZGRnIZDK6du3Kk08+yYoVK4iLi0Or1dKrVy88PT1JS0sjKyuLiooKUlJSmDx5cptkOzo6sm7dOhITE/H29r5FGc/IyODXX39l69at+Pj4tNogKZPJ6NSpE8uXL2fmzJlAk6tNSwc8tsTo0aPZs2dPq9rQHLa2tri7u2MwGMSg1M8//xwXFxdef/11kxhoTKI11tfXc+LECaKiopg0aRKPPvqo6Mt78OBBMjIyMDc3Z/r06aYQdwsLFy5Ep9Oxfft24uPjxcEVmpTY/v37myRq/I+o1WqOHj0qBq76+voya9asW3JRmorBgwfj5+fHtWvXgKbAOr1ez5UrV/jggw9Yv349kydPbvPHdr9otVqOHTvGpUuXRB+uXr16MXr06Fa7zBj974zH+sbExFBYWIi/vz+Ojo4kJSVx+vRp4uLiKCgoQK1WI5PJKCws5NChQzg5OdG7d2+++OIL+vXr16pBsKCggJiYGFJTUzEYDMhkMpRKJY6OjgQGBvLYY49x+fJlvvnmG/785z+3euFhMBgoLCxk1apVHD16lIqKCvz9/Zk6dSpTpkzBysoKg8HA1q1bb/utUXn5Tz9hzYidnR0fffQRY8aMuUUBUiqVDBkyhBdeeMHkyviMGTM4duwYn3/+OTqdDgcHB0aPHs3zzz9/y0mXxtSO3bp1axcf6sbGRhISEvj8889Fg4LxtMi2BtndD0YLupeXFwqFgsrKSmpra02ijLu7uzN+/HiqqqoICwsTT8v79ttv2bp1KydPniQhIcEkpyGqVCqKioq4dOkSO3bsEF1voGl8MQZ8yWQyLCwssLe3x8zMjPPnz3PixAksLCyYP38+y5Ytuy93uw8//BBbW1siIiLE8cuIMROQ0WUEmnaCevfuzdy5c5k9e7aY1act7obm5uZ4enri4+PDM888w4ABA247tfXcuXN8+OGHHD58mKNHjzJnzpz7zpR09epVVq1aRUVFxS1jkFwux9LSEktLS6ytrfH398fGxoasrCzS0tLEXfKxY8eydOnSVvfTiEKhuGUhYWtrK+74GAwGFAoFHTp0ENPemhpjbElubi4WFhYEBAQwe/ZslEql6H7n5uZmsuxAKpWK/Px89Ho9SqWSpUuXiuOS8TlMnjyZDh06UFFRQVZWFubm5m2KRRAEgYaGBpRKJV27dm12J6VHjx4kJyezefNmYmNjTeId0KNHj1b/NigoyKRJO0pLSzl37hwajYaQkBAUCgVnzpzhk08+4ZFHHjFJTECblfHGxkYSExNZsmQJgiCwcuVKcQBzdnbmk08+IScnp91yNstkMhYvXszgwYOZP38+KSkp4jV3d3eT5FJtjoiICHbu3Mn169dxc3Nj+vTp7ZbKz7havPkFFwSBIUOGsHDhQmJjY5sNdmtvkpKSWL16tWi5tbS0xMvLq9XWWo1GQ25uLlu2bGHFihV3LW8M3LCzs6N379707t2bVatWtVmBqaqqEoNgjPWPHTuW559/npCQENGy8/XXXxMdHd1qZbywsJDnn3+ekydPYmNjw6RJk3j++ecZOHAg6enpfPPNN+zevVvsq1wuF30hi4uLSUlJwcPD44H4jDeXvqqtGCcQMzMzXn311VsCkB8E3bt3Z/Xq1RQXF+Pq6ioefPPHBU59fT05OTlMnz7d5FmZNBoN2dnZREVFiYq4paUlf/nLXxg7dmy73w9BENDr9eJZEatWrWLv3r0YDAa6dOlist1MZ2dnduzYcdvfR48eTUJCAo2NjSZJW6nVatmyZQs7d+7k5MmTCIIgug706dOH8vJyzp49S11dHVZWVgwbNoy33noLHx8f1qxZw44dO2hoaODSpUu3BFreC25ubnz22Wc8++yzt8V9GDO3fPzxx+Lf+vfvz4YNG+jZs6fJdlN79uzJ+fPn71jG29tbXGCp1WoqKyvvWxlvbGykvLxc/FaMwXo2NjY88sgjhIaG4uTkxIABA6isrGT9+vVkZGTQ2NiIg4MD8+fPx8bGpnWdvAcMBgOlpaUcP36c8+fPiwF+7Wm8sLCwoEePHiiVSgwGA++++y56vR5XV1eTjBsajYbk5GTxnAWFQiEmLLg5gcAvv/xCVVUV9fX14oKotYcPCYJAeXk5e/fuJTQ0VFTEjfnDjf+MxowOHTqIsWMPE1Mc4tUcHTt2ZPXq1fTq1Yt169axYsUKXnvtNX788cfbdgvulzaNAAaDgezsbNavXy86999sSSgoKLglzWB74uDgcNvqs3v37uJWh6nZtGmTmFbIz8+PkJCQdsvaotfr0el0mJmZoVAoaGhoIDc3l/j4eHbv3v1QrKO1tbX84x//IC8vD51Oh4WFBY8++igvvfSSuLV/P+Tm5rJ9+3Y++eSTe54Evb29efrpp3njjTewt7c3mfUhICAAb29vFAoFCxYsYObMmfj5+YmTh0ajwcXFRXwWraGwsJBnn32WmJgYOnTowAcffMCoUaP4/fffeeONN6isrBR9S40HSHXt2pVr165RWlpKTk4Or776Ktu2bWvzyX13w7gD4uHh0aY0UTejVCoZOXKkmAHnf//3fx+oIm4kKCjojtcFQSAyMpKcnBymTp1q0viT+vp6IiMjmT9//i0pHFevXs3kyZMfSO7zrKwsNmzYQEJCAomJiWI6uMcff5wXX3yx3TNRmRKNRsPSpUtZv3490DQnvP3228ycOZPu3btTUFDA3//+d44cOYJcLsfT05MVK1aIisqPP/5IcHDwPRkC7kRgYOBtY2BDQ8MtC1pLS0vef/99/Pz8HvgJzevWrSMqKgo7Ozv8/f0JCAhodV3GbGVLlizBz8+P+fPn3zYfpaWlkZ+fT319Pfb29gwbNqzdT2hOSEjgr3/9K3v37sXMzIzevXubJAPTnbCwsGDo0KEIgkB4eDg7duxg+vTpjBgxwiQJLHJycoiOjhYDG29m8ODBuLq6kpeXJxrHlEolo0aNYuHCha3OMpabm8uyZcvYs2cPM2fOFDPS2NnZUVRURGlpKYmJiVy9epXS0lKCgoJMsrvVVtatW0dNTY3JXaPNzc3x9vbG2tqa9957j1OnTonuWs7Ozm0aL9s0CqSnp7Nu3TrCw8OZOXOm6IaSn5/P66+/TmRkJA0NDbz44osMGDCgLaJahUKhaJdgq8OHD5OamkpDQwODBg3ihRdeYMqUKSaXY2T+/PlcuHBBVMZVKhWlpaWoVCrRB+9B57t+8803+fnnn8VgBnd3d2bMmNHqFXhCQgKbN28WFXFzc3P69OnDgAEDmDt3Lnv27GHz5s3U1NQgk8kYMmQIb7zxBrNnzza50qJUKnnnnXdwdXUVD2W5Gb1eT0lJiWipuPmEsXvlwIEDxMTEoNVqGTZsGAUFBfz5z38mLi6O0tJSsZy5uTkeHh7s2rWLvn37cuTIEZ544glqa2uprKykvr6+VfLvhXHjxmFhYYFareb48eOEhYWZTBk3MzOjR48enDlzRrT6/ztiMBj45ZdfmDlz5m2ZV9qC8XyAiIiIWxRxV1dXJk+ebDIFrbCwkLi4ONLS0rCzsyMwMJDTp09TVVXFwYMHxdzMN7tUeHl54e/v3+pMKjqdjqtXr/I///M/uLm58dNPPzVbTq1WExsbS2JiokkybhkMBq5evYpMJsPZ2ZmtW7cyYsQIrKysyMjIYMOGDezbtw+lUklQUBCfffbZbePV66+/ztChQ1EoFOLx3u1Br169CA0NfaAL0MLCQpYvX054eDiVlZU8/fTTLFmypFV1WVlZERQUxLRp01i0aBGenp5A8ylQDx48KC6APDw8eOKJJ9ot81d5eTmHDh1i48aNnDx5EgsLC/r168dvv/1msrHrThgMBi5fvszChQuRy+UsWbLEZAva0tJS0fUFmhbzY8eORSaTiQY7I5aWlsyePZvFixczdOjQVss07j5rtVp27twpWuUNBgNyuVxcYAYHBzN37lweeeQRk6c1/vzzz2lsbGTBggW4u7vfsWxZWRnPPfccR44coWvXrixevLjVp43fC7/++it9+/bl/fffp0+fPm2KlWv1iN/Y2EhSUhIHDx4kODiYzz77DJVKxffff8/WrVtJS0vD3d2dxx9/nGeeeeaBpLprj+Cq5ti7d6+oMPXp04e+ffu2q3V65MiRJCQkiKecwb8+hs6dO/OnP/2J7t27t5v8m6mvr+fMmTMcOnRIdI0xNzdn9OjRbXrGOp1O3EWxtbXlueeeIywsDDMzM7777jsOHjxIXV0dlpaWzJs3j4ULF9K/f/92sx6GhYXdMajMuD2XlJTUqvqNv4cml6fIyEgMBsMtO0leXl4sW7aM+fPnY2tri1wuZ9y4cbzzzjusWbOG6upq/vrXv6LRaJg8ebLJF543H1ZRVFTU6l2AO2HqwzBMiXFijY2N5Z///KdJT4XMzc3lwIEDREREiH9TKBR8//33+Pn5mWw82bFjB59//jmlpaWiS5Dxu9XpdOh0uluOrg4MDOSll15i5syZrT5xtbS0lIiICE6ePImLiwt5eXmismbEYDCwbds2IiIiUCqVDBgwwCTxNsa+9OjRg4EDB2JlZUV9fT2//fYbe/fupaSkhOHDh7Ny5cpmlRSZTCamwGvPd/OPR9W3N/Hx8bz55pvEx8dTX1/P008/zWuvvSamt7xf+vXrx+bNm/H3928x80xxcTEbNmxg9+7dqFQq/P39efnl0ANDjQAAFGFJREFUl5k2bVpbu3MbWVlZ7N27l4iICJKSkiguLhbjTTZu3Hjb+2dKjCfn1tTUsHr1aqDJ1XH16tVtyrD1R4KDg1m0aBEFBQWcO3cO4Db3VHNzc8aNG8e8efMYPXo07u7ubXrPXF1dWblyJWFhYVy9epXExER69OiBmZkZDg4OjBw5kh49eojxDqYMVjWyadMmcnJyOHjwID179mTMmDFibnqjIUqtVrN+/Xr2799PQ0MDGo2GcePGMW3aNFxcXEzanpvp0KEDZmZmlJSUUFRUhI+PT6ufd6uUcYPBwLlz59i8eTNlZWU4Ojpy4sQJdu3axcmTJ6mqqiI0NJS3336bwMDAW9LStCcPanAzBgANGjSIiRMntnpAu1fmzp3LxIkTSUtLIycnR1TYlEolgwYNarOv0r1gzNe6du1a9uzZQ2lpKYIg4OHhwfPPP89jjz3WJkuSv78/48aNIyMjA5VKxc8//8z+/fsBxCChqVOnMn/+fDFPcnsemX0nJV8mk6FQKJDJZK32pe7UqRPdu3cXT0Q00rlzZzw9PQkMDGTevHn069fvljRR5ubmt1gQL126REpKCiEhISZXxmUyGV5eXmKO6vbwG/93RRAESktL+eijjxg8eDBDhgwx6TdWWFhIfHy8mCvY0tKSTz/9lLFjx5p0rAwNDcXKyoqUlBROnTrVoj+xlZUVAwcOZNmyZYSEhNyWuux+sLOzEyfokpIS3nnnHf72t7+JeYHz8vLYuHEju3btIjs7mzFjxjBjxgyT9tu4i3jq1Kn/1969B0Vd/X8cf+5uulzkEiwsF7mp3GTFXcUM0tJSKcKEKVGoLLpQFlOZWmMzTVOTOZVdvt8u3++Y1WQ3raHrlKN5SRFFszGzVCAlHBVEyY0FMmHh+4ez+9PK6gf7+RzS92OmmfAPX3s+fs7ZN+dzPufw0ksvUVVVRWtrK7m5uZSXl5OVlXXWpw/9+RfEs+no6GD37t1UV1czYsSIM5ZkLF68mKVLl9LQ0MCJEycoLS2lrKyMkSNH9vrfODg4mPT09D8dgzs6Oti2bRv19fX09PQQGhrK0KFDfTaBcuzYMXbs2MHGjRtZv349+/fv9+5hPnToUIqKiiguLiYxMVGz2sBkMmGz2bj11lt56aWXaGxsJDAwkAcffJBbbrnFp2Oyv78/DoeDsrIyXC6Xd0MHOLXTyaJFixg8eDDJyclERkZ6t3vsiwEDBhAfH09hYSF5eXneFzk9TzMDAwMxm82a1h8zZ87k2WefZfv27ezatYuVK1f+brmg2+3m+PHjOJ1ODAYDEydO5M477/TZYUdms9m7j/26deuYMWMGAFu2bMHlcnnfu+nLhHCvinG3201NTQ2bNm3yHm+7aNEiDhw4QGBgIGVlZZSUlGCz2f7whSg9+Pv7a3oIDpxaI6bHwO3ZBiw6Opr29nZvUWQ0GgkLC9P8M3i2b5o/fz4bN26kqakJgMTERIqLiykqKiI5OblPj1yTkpIoKSnh119/ZcuWLdTU1Hh3QJg2bRr5+fk4HA6GDh1KcHCw0mUNAwcOZMSIEX16cXLcuHE888wzrF279ozHizabjfT0dMLDw0lISPjLwTw6Ohqr1arZXvoOh+MP1yie69ra2nj//fdpaGjg2Wef9emhZQcPHuTLL79ky5YtdHZ2EhwcTElJCdddd53PX2pLTk4mOjqa1tZWCgoK2LdvH3v27GHr1q3ew2dCQ0MpKysjMzOTjIyMPh9OEhAQwMUXX8xTTz3l3bXD5XJ5l70cO3bM+0SpsLCQ6dOn43A4+tZQTs1Qzpgxg8bGRn744Qfmzp3L7t27+f7773G5XIwbN46ZM2cyduxYzfqLKt9//z3//e9/2b17t/dY9OrqaqqqqtiwYQP19fVkZWUxY8YMJk+e3Ofx2mg0/uVkyI8//sjRo0c5efIk4eHh2O32Xi9XbW9vZ+vWrRw5coSYmBgqKyvZuXMnDQ0NHDlyhObmZvz9/cnJySEnJwe73Y7dbicuLk7z74r4+HimTZvGxx9/TFNTEyNHjvSeYeDrJYRhYWHk5eWRnJx8xrtVfn5+2O12/P39//S4+d644IILlB4sd+ONN3Ls2DHeeecdjh49+rt3yk6fIU9OTiY7O5vZs2eTmprqs1/8QkNDcTgchIeHs2TJEiwWC0OGDOFf//oXLpeLsWPHEh0d3ae8XhXjRqORxMREcnJy2L59u/eN4YyMDCZOnEh2djbp6elKZxhCQ0OxWq2a/CJgt9tZs2YN+/fvZ+fOnYwaNUrTdUkeAQEBPi0K/o6Ojg527drF0qVLWblypXcLw+joaO69914mTZrU54EdIDAwEIfDgb+/P1OmTPEW/HBq1wHPvq39YSs/o9FIdHQ0N998c6+3y7JarVx++eUkJSWdseuC1Wr1HtBxNuPHj2fhwoWcPHkSi8XCRRddpMm7EXCqGF+9ejUWi0VZAeNZtxgcHKzJ7HxHR4d3T2Q49cVfXV3Nhx9+SGFhoc+3K62trWXXrl3ecTMhIYGysrJeLwv5M54v54iICBITE7Hb7TQ2NjJhwgRaWlqAU31v/PjxhIWF+eRL3HNAWH5+Pk1NTSxZsoRVq1Z5i4XOzk7S09PJzs5m4sSJ2Gw2n3zZm0wm8vLyCAkJoaWlxbvL0rXXXgvAsGHDGD16tOaTNH9HamqqTwum48ePU1NTw3fffcdbb73FqlWrqKuro66ujvDwcIqLiykoKGD8+PFERkbq8t28bds270vooaGhJCYm9vp01ebmZp566ilaW1sJDQ2lrq6OlpYWQkJCvEexDx06FJvNRkpKCpGRkZru1nI6Pz8/Ro4cSXl5OW+88QaHDx9m06ZN3gPDfHmtPWcC9OXk1H+ahIQESktLCQoK4uuvv2b//v00NjbS09NDRESE95rYbDbGjRvHsGHDyMrK8mn/GjhwIEOGDGHSpEmsWLGCxYsXExkZydq1a3G73RQUFPT6tG+PXhXjJpMJh8NBeXk5W7dupbOzkzFjxngLA73fDodT64xTU1NpaGjg559/PmMNsq9dfvnltLS00NzcTFJS0j/ysebf5XK52Lx5M8uWLePkyZMYDAYyMzPJy8vj+uuv/8vC8f8jKCiIMWPGaLJXu6+ZzWbKysqorq7udfv9/Px6tZOBw+HwyUzi33HJJZdw2223MXjwYE2Kxb/D8wRo1KhR/PDDDz5fA9jQ0IDL5cJisTBo0CBqa2tZvnw5ISEhlJSU+PwX4Pr6eurq6jCZTN4nQnr8e5pMJoKCgry7aGjJYDBw4YUXcsMNNzBgwABqa2u9L2l6tmrNzMzs82zSbzPj4+M1P424r9LS0pgyZYpPvzesVisZGRl8/fXXrF+/nkGDBhEZGcm0adMYMWIEEyZMwOFwaLq077fq6+tpa2sjJCSEzMzMXp/7AP+3DMHlcuF0OomKiuKSSy4hLS0Nu91OUlISMTExDBo0SMmEjcViYdasWfj7+7Nnzx6ioqKUrQo4F40cOdK7E09dXR2HDh2ip6eHqKgoLrjgAqKiosjMzOzTzkB/JSwsjKKiIhobG/n888+9y5WvuuoqcnNz+zyh0Ouq2WKxkJubS25ubp8+gK9EREQwY8YM2tra+Oqrr3C5XL87eMFXbDYbNpvN539vf/TLL79w5MgRTp48idFoZPjw4cyePZvp06freiBJf2M0GklPT9e08/cHWVlZ3hfaVDEajcTFxTFr1iy2bdtGRkaGT/9+p9PJ9u3bMRgMBAYGsnPnTn788UfmzZunyROvzs5O2traaG9vJzo6mltuucXnGf2BZ/eM+++/X/VHUc5kMmG1Wpk8eTK5ublMmjTJp8V4amoqM2fO5KeffqK1tZWIiAjS0tK44447ej0b3VdpaWne9w+uueaaPm15FxERwZw5c2hsbMRgMJCUlMSoUaPO2F9bJc/T0nvuuUf1RzlnJSUlkZSUpCw/ICCA7OxsgoOD6enp8Z5F8tBDDzFs2LA+T0IbnE7nWavVvq4d1Ft3dzcbNmzgrbfeor6+nry8PObOndsvOus/1aFDh3jnnXd45ZVXMJvNPPzwwxQVFan+WEL4THd3N59++imfffYZDQ0NmEwm7rnnHq688kpN8qqrq3nxxRdZs2YNeXl5vPbaa5rkCCGE6F+6urpob2//3Z+fU8W4EEL0d263mw8//JCXX36ZtLQ0Xn75ZdUfSQghhA6kGBdCCCGEEEKRsxXjf7rIpbu7+7zaW1gIIYQQQggtnG3HlT8txj3b2AkhhBBCCCF8T93JKUIIIYQQQpznpBgXQgghhBBCESnGhRBCCCGEUESKcSGEEEIIIRSRYlwIIYQQQghFpBgXQgghhBBCEU2K8YaGBqZPn05CQgIpKSnMnz+frq4uLaL+0L59+7BarZSVlemSt2TJEiZMmEBkZCSzZ8/WJdPj6quvxmq1EhsbS2xsLFlZWbrk1tTUMHXqVOLj43E4HHz66aeaZ/7666+Ul5djs9kYPHgw48aN44svvtA893R63lsq26vynvbQux+r6kuePM9/YWFhzJ8/X5dsVW0uKysjNTWVuLg4Ro8ezbJlyzTPPN/GD1BznU+nd3tBbZsrKiq46KKLiImJwW63s3nz5nM6V9X4oSpXy9r2T/cZ76158+ZhsVioqanh559/prCwkKVLl3LnnXdqEfeH+aNGjdIlCyAqKop58+axbt06fvnlF91yPZ5++mlmzZqlW15XVxclJSWUlpby0UcfsWnTJoqLi0lPT2fYsGGa5sbGxvLZZ58RFxfH6tWrKS0tpaqqioSEBM1yT6fnvaWyvarvadC/H4P+fQng0KFD3v9va2sjNTWVgoIC3fJVtHnOnDm88MILmM1mamtryc/PJzMzE7vdrlnm+TZ+gJrrfDoVfVhVm9evX88jjzzC66+/zujRo2lqatI0T3Wuh4rxQ1WulrWtZjPjhYWF+Pn5YbVaueKKK9i7d68WUb9TUVFBSEgIl156qS55ANdccw35+fmEhYXplqlSbW0tTU1N3H333ZhMJi677DLGjh3L8uXLNc0NDAxkwYIFJCQkYDQaufLKK4mPj+ebb77RNNdD73tLZXtV39Mq+nF/8Mknn2CxWMjJyVH9UTSVnp6O2WwGwGAwYDAYqK+v1zTzfBs/QM119lDVh1W1edGiRTzwwAOMGTMGo9FITEwMMTEx52zu+UjL2laTYnz27NlUVFTQ0dHB4cOHWbNmDVdccYUWUWdobW3liSeeYOHChZpn9SePPvooQ4YMITc3l8rKSiWfoaenhz179uia2dzczL59+0hPT9c8qz/cW3q2VyWV11p1X3r33XeZOXMmBoNBt0xVbZ47dy7R0dGMGTMGq9XK5MmTdcuG82f8UHGdVY+XerfZ7XazY8cOWlpacDgcDB8+nPnz52v+VFFV7ulUjR8qcrWsbTUpxnNycti7dy9xcXEMHz4cu91Ofn6+FlFnWLhwITfeeCOxsbGaZ/UXjz76KN988w179uzhpptuori4WPNZgOTkZCwWC//+97/p7Oxk3bp1VFVV6ToAdHZ2cvvtt1NcXExKSormearvLb3bq5Kqa62iL53uwIEDVFVVUVxcrFumyjY/88wzHDx4kJUrVzJ16lTvbKYezqfxQ8V1Vj1e6t3m5uZmOjs7+fjjj1m5ciWVlZV8++23LF68+JzM9VA1fqjK1bK29Xkx3t3dzbXXXsvUqVM5fPgw+/fvx+l08sgjj/g66gzffvstGzZs4K677tI0p7/JysoiKCgIs9lMSUkJY8eOZfXq1ZpmDhgwgLfffptVq1aRkpLCiy++SGFhoW6Pxrq7u7njjjsYOHAgTz/9tOZ5qu8tvdurksprraIvnW7FihVcfPHFJCYm6papus0mk4ns7GwOHz7Mq6++qkvm+TZ+gL7XuT+0F/Rts7+/P3Dq5dGoqCjCw8O56667NO9LqnI9VI0fKnK1rm19/gLn8ePHOXjwILfffjtmsxmz2cz111/PwoULeeyxx3wd57Vp0yYOHDiAzWYDoL29Hbfbzd69e9m4caNmuf2NwWCgp6dH8xybzcbnn3/u/XnKlCm6zOj19PRQXl5Oc3Mz77//PgMGDNA8U+W9paK9KvWnfqxXX/JYvnw59913n255f0TvNnt0dXXpMrN1vo0fv6XHde5P7QV92hwaGkpsbOwZy8v0WGqmKvdsVI0feuRqXdv6fGY8PDychIQEXnvtNbq6unA6nbz77rtkZGT4OuoMN998Mzt27KCyspLKykpKS0uZMmUKH3zwgaa5cKqznzhxArfbjdvt5sSJE7ps5eh0Olm7dq0377333mPz5s1MmjRJ8+zvvvuOEydO0NHRwQsvvEBTUxMlJSWa595///3U1tayfPly76yA1lTeWyraC+ruaVXXWmVfAti6dSuNjY267qKiqs1Hjx6loqKCtrY23G43a9eupaKigssuu0zTXDi/xg9V11nleKny3iopKWHJkiUcPXoUp9PJf/7zH3Jzc8/ZXFXjh6pcrWtbTbY2fPPNN1mwYAHPP/88JpOJSy+9lCeeeEKLKK+AgAACAgK8PwcGBuLn54fFYtE0F05tsfPkk096f37vvfd48MEHWbBggaa5XV1dPP7449TV1WE0GklJSeHtt9/WdHtBjxUrVrBs2TK6urrIzs7mo48+0nxd3oEDB3j99dcxm82kpqZ6//y5556jqKhIs1xV95aq9oK6e1rVtVbZl+DUi5v5+fkEBQXpkgfq2mwwGHj11VeZM2cOPT09xMXFsWjRIvLy8jTNPd/GD1XXWeV3sao2AzzwwAP89NNPjB49Gj8/PwoKCpg3b945m6tq/FA5VmtZ2xqcTqf+zxSEEEIIIYQQ2uymIoQQQgghhPhrUowLIYQQQgihiBTjQgghhBBCKCLFuBBCCCGEEIpIMS6EEEIIIYQiUowLIYQQQgihiBTjQgghhBBCKCLFuBBCCCGEEIpIMS6EEEIIIYQi/wNfNuZFHdd3CQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "\u003cFigure size 936x216 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuMAAABQCAYAAACzvHtWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydeVhT19b/vwkJCXMAmWRWEBEBFRQRRaxapWqtWmutHazVqh3s6K3trdrb9rW3rdbWtlatt7Zaa52rVgTFARVQEZBB5plACEMIEMic9fvDy3mlAjIEvff9nc/z5BFz9tlr73Ny9ll77bXX4sjlcgILCwsLCwsLCwsLywOH+7AbwMLCwsLCwsLCwvL/K6wyzsLCwsLCwsLCwvKQYJVxFhYWFhYWFhYWlocEq4yzsLCwsLCwsLCwPCRYZZyFhYWFhYWFhYXlIcEq4ywsLCwsLCwsLCwPCVYZZ2FheWg0XL+BixOnMP+/Gj0HDddv9Khsb7m9/iMUffdDn8/vKb3p01/pTdkHRZu4CrG+I2DQ6Qak/N0oq6txLjgEpNf3+JxLUdNQn5jUo7KxviPQWl7e63b191wWFhaW7uA97AawsLCwtDPxzCmj1CM+ehziw0cx/vdfme8CPvnIKHX3lt706e6yhdu+Q1t5BYK3fDEQzTIal6KmYeT/fIxBERP6XZfZ4MGYnpFqhFaxsLCw/PfAWsZZWFhYWP7jISKQwfCwm8HCwsJidFhlnIWFpV+U7NyN9Nfe7PBd7iebkPPx/wAAxEeO4cqM2Tg3KhQJUx5FxYGDXdZ1t8uBXqVC5t8+QHzIeFyZORvNmVl/kfsjEh6ZgXOjQnFl5mxIz8YDABRFxcjZ8A/I02/hXHAI4seEAQAy//YBCr76hjm/8uBhXJ46A+dDxyN15atQSWuZY7G+I1Dx2++4PG0m4seEIeejT0DUebLi+7Xz3j69f6fsjNko2fWvDi4t7WXrLl9ByY5dqImJxbngECTOmXfnWh49joQpj/77Wk5H9YnOre7yjEwkL1yM+DFhuDghEjn/+BQGjaZH/SO9Hnn//ALnx01AwpRHUXcpoavbhcx334OqWoK0la/iXHAISnb9izkmOfknLkU+gvPjJqB4+w7mezIYmHt3fmw4bq15Cxq5HMC9Li7Xl7yAgq++xrVFS3AucAzaKiu7bEtP+g0AdZcuI2HKozg/bgLy/vllBwVffPgorsyYjfiQ8Uh5cQWUVVWdyqm7lIArM+/8pi9OjELp7p+6bRcLCwtLd7BuKiwsLP3CZXY0ir7bDp2iFTxLC5Bej5ozsRj9/bcAAFN7e4Ts2g4zD3c03riJm8tXwiYoEDYBI7qtt+jb7VBWViDyfCz0SiVSX1rZ4bi5hzvCDuyDwGEQas7EIfPd9zApPhaWPkMx4uON97ip3E1D8jUUbNmK0D27YeXjg7zPv0DGm+8g7MA+pkzdxQSEHzsEnUKBpCcWwuGRKDhETup1O+8pK67C5Atx0LcpcXPFqk7LOUROwpBVL3dwU9G1tSH3k00IP3YIlkO8oaqtg7ZJ3un5HBMT+H+wDtaBAVDVSJH60kpU7P8dXi8+f9/+VR48jLqLCZhw4ihMzMxw6y8TrbsJ2vw5ZDdTO7iptInvKLCNqWmYdDYGraVlSF6wCE6PToelz1CU790P6bnzCNv/C0zt7JDzyf8g56NPMerrzZ3KqP7jJEL+tRMW3t5AFxOi3vS79tx5hB8/BH1bG1JeeAkWQ7zh/tSTkMafR8mOXRizczvMvTxRunM3Mt5ai/GHfrtHTvYH6xH8zVewGxsKbVMT02cWFhaWvsBaxllYWPqFmasrrANGQHrujmW6Ifk6uEIziEYHAwAcp0yGuacHOBwO7MLGYtDECWhMub9fcE1MLIasXglTkQhmLi7wfP7ZDsedo2dC6OQIDpcLl1nRMPf0QFNmZo/aXH3yT7gtmA+bgBHgCkwx7J23IL+V0UGpGrJyOfjW1jAbPBj248ehJTevT+28t+zL4NvYQOji3G3ZzuBwuVAUFEKvUkHo6AArX99Oy9mMDIBodDC4PB7M3Vzh/vRTkKWkdCjTVf9qzsTB84XnYObiAlORCENWruhVG9vxee0VmAiFsPYfDmv/4WjJywcAVB44iGFvvwGhizO4AlP4rHkN0rizXW74dJ0/D1a+vuDyeODy+d3K7Em/vV9+6c69GjwYXkufh+TP00y7hqx6GZY+Q8Hl8TBk9ctozs3r1DrO4fGgKCqGrkUBvo3NfSeWLCwsLN3BWsZZWFj6zeA5syD58zRc582F5NSfGDxnFnOsLuEyir7djrayMpCBoFcpYTVs2H3rVNfWQujizPxf6Dq4w/Gq4ydQ9tPPUFZVAwD0bW3QNnZuKe6sbuu7FCiehQX4IhuopVKYu7kCAEwdBjHHuUIz6Frb+tTOv5Y1c76r7F3n3Q+euTmCv9mCst17kP3BetiOGQ2/9/8Gy6FD7inbWlqGvE2foyk7G3qlCqTXw2ZkR4Wxq/7d6Y9Lj/rTHR3rF0LXdqd+ZXU10l5ZAw73f21BHC4XmvqGTuvpzTXqSb873KvBg6GW1t1pV1U1cj/dhLzP7towSwSVtBZmrq4d6hj93Tco3r4DBZu3wspvGIatfRu2o0f1uJ0sLCwsd8Mq4ywsLP3GeeYM5H32BVSSGkjPnWeW9g1qDdJfexNBX3wGx2mPgMvnI231a136X9+NwNEBKkkNY/1VVUuYY8qqKmT/fQPG7f0JotGjwDExQeKceUy9HA7nPnU7QvVvJR644wKilTdB4OTU6753185Oy9ZIYenrc6espKbLsp31wWHSRDhMmgi9SoXCrd/g9ocbEHbgXlec2xv/AesR/gjeuhk8SwuU7dmLmrizPeuPgwNUkv/tQ3f96aqd3SF0cUbgZ5/CNmTMPcc6dffoRf096fdf75XAyeHf7XLB0NUrMXjunPvKsQkKxJgd38Og1aLi19+QseZtRF250ON2srCwsNwN66bCwsLSb0zt7WAXNhZZ6/4OMzdXWPoMBQAYtFoYNBqY2tmBw+OhLuEy6q/2LCa0c/RMlOz4EdqmJqgkNSjft585pm9TgsPhgG9nB+DOJlFFYdH/tmeQPVQ1Nfds3mvHZfZjEB89juacXBjUGhRu+Ro2wUGMVbw3dNfOTsvu/HfZGikquilram8PZVU1s8FQXV8Pafx56NrawDU1hYm5OcDpfAjXt7aBZ2EJEwtzKIpLUHHg9171p3zvr1BJaqBtakLJrh+7LW9qbw9lpbjH9XssXoSCr75h3D80DTJI48/3+Pzu6Em/S3f/BG1TE5QSCcr37oPLY9FMu0p2/oiWwkIAgLalBTVnYu8536DRoPrEKWhbWsDl88GztAS4vZuQsLCwsNwNq4yzsLAYBZc5s9GQlNzBRYVnaQH/9R/g1htv43zIeEhOnYbj1J4l7vF5/RWYuQ5GwpRHkfLicrjeZbG09PWB10tLcf2pxbgQPgktBYUQjRnNHLcfHwYrHx9cnBCJ8+PujX89KGICfN98HemvvYmLEZFoq6jEqK2dbyDsTzv/ytDXVkPo7HSn7AsvwXnmDHBNTTst6xw9EwBwfuwEJM1dADIQyn76BZcionA+NByyGzcR8I8NnZ7r995aSP48jfhRobj94QZG4ewJbouexKBJE5H4+DwkPfEknB6d3m35IatWoHj7DsSPCetRVBHPF56D49QpSHlxBc6NCkXywsVoyuiZr//96Em/Hac+gqQnFiLp8flwiIqE28IFAACnR6fB++XlyHjzXZwbNRaJj81FXcKVTuVUnziFhKjpODdqLCoOHPyPjwXPwsLynw1HLpfff72YhYWFhcXoVOz/HZLTMQj7be/DbgoLCwsLy0OCtYyzsLCwPCBUtXVoTE0DGQxQlJSi7Kc9cJo+7WE3i4WFhYXlIcJu4GRhYWF5QJBWg9vrP4JSXAWetRVcZkXDY8nTD7tZLCwsLCwPEdZNhYWFhYWFhYWFheUhwbqpsLCwsLCwsLCwsDwkWGWchYWFhYWFhYWF5SHRrc+4jY3Ng2oHCwsLCwsLCwsLy/9ZDAYDWlpa7vmetYyzsLCwsLCwsLCwDDCGfydx+yv/9dFUDh8+DJVKhfLycqSkpKC5uRlBQUEYO3YsgoKCEBQU9LCbyMLCwtIBg8GA9PR0fPvtt6itrcWECRPw4YcfPuxmsbCwsPSIsrIyHDt2DImJiVAoFJg+fTqeffZZODs7G11WTEwM9u3bh1mzZiE6Ohr29vZGl/Gw+a9VxokIu3btws8//wydTge5XA6xWAyVSoW8vDzcunULs2bNgpeXF6ytrR9Yu/R6PcRiMTZt2oRFixbhkUceGRA5RAS5XI6TJ08iMTERHh4eWLVqFQYNGjQg8lhY1Go1mpuboVarAQA8Hg88Hg/Xrl2Di4sLnJyc4Obm9pBb2XtqamogEAjA5/MhFouRk5ODyspK+Pn5gcPhYMiQIfD19TWaPIPBgLy8PBw8eBAnTpwAANZo8IAoKSnBuXPnkJ2djaCgIKxYsWLAZWo0GpSXlyMzMxO3bt2CVCoFEYHD4cDV1RXOzs4IDw+Hj48PzM3NB7w9/xdQq9VobGzElStXoNPpEBwcDG9vb5iZmT3spv1/w9GjR7F//34UFBQAALhcLnx9fTF37lyjyzpy5AhiY2MREBAAjUZj9Pr/E/ivVMb1ej1Onz6NH374AZmZd9Ioi0QiDBs2DFqtFhKJBCkpKbCyssLkyZMRFhb2wNqm1WqRk5ODPXv2IDw8fMDk6PV6VFVV4YcffkBKSgpcXV0RHh6OqKgomJiYGF2ewWCARCJBTExMty+wpqYmtLW1wcbGZsBfLHK5HJmZmSgoKMCECRPg7+8PDodjlLpVKhUuX74MABg6dChcXV0hFAq7PUej0aCmpgb29vawsLAwSjv+E6ioqEBZWRmKi4tRXl6O5uZmAIBQKISNjQ0SEhLg4OAAe3t7fPXVVw+5tT1Hq9Xi6tWrSExMBBGBz+ejoqICeXl5qK6uho+PD4A799/X1xdTp07FsGHD+v18VVZW4vTp04iPj4e7uzsiIiIwc+ZMY3TpvwKNRoOGhgZUVVUhJycHtbW1sLKygru7O+zt7REUFDRgSlVxcTEOHjyIpKQkjB07Fo899hhcXV2NKqOiogLV1dWoq6uDXC5HW1sbMjIykJeXh9u3b6Ourg5BQUHw9fWFr68vLCwswOfzweWyXqM9QaPRICkpCbGxsbhx4wZ0Oh1GjBiB8PBwREREwNvbGzzef45qEx8fj0GDBsHHxweWlpYPuzlGQSwW4+zZsygsLISbmxtGjx4NX1/fAdE9FAoFUlNT4evri+HDh/9HXcPy8nIUFRWhsLAQra2tIOoYKdzZ2bnHlvz/nF9sDyEi1NXV4csvv0RmZiasra0REhKCIUOGICgoCGq1GjExMUhOTkZ5eTlu3779wJXxgoIC8Hg8DB8+fEBkGAwGNDQ04NKlS0hNTQUAxqIXGRk5IA+ERqNBSkoKNm3a1K0yXllZiZycHHh5eWHcuHFGb0c7er0e2dnZ2L59OxISErB8+XJs3LjRKIOwTqfDuXPnsH37dohEIjzzzDNwcnK67zkXL17ErVu3EB0dDX9/f/D5/D63gYigUChQUlICpVKJsLAwo000ekJbWxtKSkpQWVmJ1NRUpKWlITc3F2KxGK2trQAAgUAAkUgEmUwGrVYLANiyZcsDbWd/uH37Nn744QckJCSgpaWFsY4DgImJCVJTU0FEiI+Ph0AgQGVlJd544w04Ozv3S3EqKirC5cuX0dDQgEWLFmH16tXw9vY2Vrf+YyEilJSUIDc3F/n5+bh9+zYSEhJQUlICBwcHjBw5El5eXli2bBkmTpw4IG2QSqUoKiqCWq1GQUEBUlNTja6Mx8fH48KFC6isrERDQwP4fD7a2tpgbm4Ob29vREREYPr06Rg3bhyCgoL6NU50BRGhoaEBEokEEokEKpUKVlZWcHJygqen53+tscBgMCA7Oxu7du3CoUOHGOUnKSkJSUlJKC4uxqxZszBixIgHuiLeHbt378aYMWNgb28/4IqkSqWCWCyGTCaDo6MjXFxcIBAIjCpDq9XiyJEjyMvLg6WlJebOnYv58+fDycnJ6LIAID8/H3V1dViyZAlGjx4NKysro8voKXq9HhKJBFKpFLW1tbh58yaSk5ORnZ0NHo8HGxsbWFlZQaFQQCwWY/DgwXB1dcWUKVPuW3efNBe1Wo2mpiYolUqYmJjAxMQENjY24PF4MDU17UuVPabd8nzt2jUIBAJERkZi69atcHZ2hpmZGeRyOWQyGXJycgDggSoGRITm5mbcunULzs7OGD9+vNFlqFQq1NXV4fr169i+fXuHY21tbWhoaIClpSXMzc2NamlRqVRITExklK6uqKurw4ULFzBo0CCMHTt2wK5/c3MzLl++jNjYWLS2tuLPP//Ehx9+2G9lXK/XIy8vD++99x7q6+sRFRUFGxub+w6i5eXlWLduHVpbW+Hr64uhQ4f2+SWr0WhQV1eH9PR07NmzBw0NDYiLixuQga4rKioqsHnzZhw7dqzTGX/7c69SqYy+AqLVaqHX69HW1obW1lZoNBrweDyIRCKYmZkZZYxRq9XYs2cPLl68CLlcDjs7O3h4eMDd3R0ikQiWlpYgIuj1ehw7dgxSqRTff/89QkJCMGPGjH5FmmptbUVrayvs7OwQFhb20BRxlUoFqVSKlpYW5v5aWFjAzs4OlpaWMBgM0Ol0MDMzM8pzXFtbix9//BEHDx6ERCKBVquFqakphEIhGhoakJCQgNTUVOh0ugFTxuvr61FZWQngjmJXX19vdBlXr17F0aNHAQBeXl4ICwvDyJEj4eHhAVtb2wFzXWxHrVajvLwcV69eRXx8PDPx8/LyQnh4OFatWoVRo0b1+TnSarVobGxEbW1th3GBw+HA2toaHA4HPB4Ptra20Ov1IKIOE93+oNfrcfLkSRw8eBAAYG9vD4FAAA6Hg7KyMmzbtg1ZWVlYvnw5JkyYAFtb24dqHGhra0NmZiYGDRrEuPcNBO2rTQUFBThx4gTS09MRGRmJhQsXYsiQITAYDDA1NTXK2CmVSrFlyxbU1NRg9uzZeOqppzBmzJgBu85nz56FUqmEk5PTQ3PjUqvVqKurg0Qiwfnz53H58mXcvHkTWq0WdnZ2GDFiBCZMmIDAwEAMHToUeXl5+Pnnn1FVVQWxWNwjGX3SXG7fvo0jR44gPT0dtra2EIlEmD17NpycnJil3a5of1AFAkGfLLhEhJaWFtja2sLPzw87duyAi4sL80M4e/YsLl68iIaGBowYMQLBwcF96WKfUCqVyMjIwB9//IFly5YNiIzbt29j3759SEhIQGFhYYdjycnJ0Gq1CAwMRGRkJEQikVEeECJCU1MTLl++fN+HedCgQbCyskJ5eTlUKtWALjdnZ2ejqakJTk5OmDFjhlGU1fr6eqxduxalpaV48skn8be//Q2BgYHdnqPX6/H999+jvLwca9euxfjx4/tseSIi5OfnY8+ePfjtt99QW1sLa2trSKVSuLu7D9iAp9frGaWXz+ejtrYWv/32G/R6PQDA1NQUPB6vw+R7yJAh8PT07FBPf9qn1+uhVqtRUlLCTEauXr2KsrIyODs7Y/78+Rg9ejQz0emPda+kpAQXLlyAXC6Hg4MDVqxYgYULF8LJyQlWVlaMS5JOp4OXlxc+/fRTtLS04Mcff4SPjw9GjhzZpxcbEaG1tRUqlQouLi4PdHy6G41Gg+vXr+Ozzz7DhQsXmEn2+PHjsXz5ckRFRaGlpQVSqRQTJkzotzVKr9dj37592LZtG6OUCAQC+Pv7Mz70Go0GCoUCsbGx/e5fT7CyshqQcXrs2LFITU1FbW0toqOjH+hqkUqlQnp6Ol599VVkZ2fDwsICHh4ecHNzQ2NjIw4fPgyRSAQbGxv4+PjAYDDAxMSkV4YbiUSCPXv24NNPP4VOp2O+FwqFmDlzJjgcDpydnfHUU0+hpaUFGo0GPj4+GDJkCPh8PjOG9MVYxOFwmH1RAoEAzz33HAICAsDhcHDkyBEkJycjJiYGRUVFWL58OZ577rkB2+xHRNBoNNDr9V1OWAsKCiCTyZi2DwR6vR6ZmZn46aefcPz4cdTW1jIbxKVSKebOnQulUokhQ4Zg1KhR/ZaXmpqKpqYm8Hg8zJ8/H0OHDh2wvhkMBpw7dw4qlQp+fn6wtbUdEDn3o6CgAJs2bcIff/wB4M5v3c/PD4GBgVi+fPk9rnUcDgfh4eFQq9VYsmRJz4TI5XLq6tMVX375JTk6OhKXyyUej0dcLpe4XC5ZW1vT+PHju/1MmTKF3nrrLcrMzOyy/vthMBiovLycZDLZPd+vWbOGBg0aRBwOh6ZMmUJpaWl9ltNb6uvr6dtvvyVra2u6fPmy0es3GAz0zTffEIfDYa55+4fD4RCPxyMej0eenp508OBBamxsNIpcuVxOe/fuJTMzM1q/fn23Za9fv05LliyhiRMn3nN/jIVGo6HPPvuM3NzcCAAFBAQY7XrHxcWRlZUV8Xg8io2NJbVafd9zMjIySCgU0qxZsygnJ4d0Ol2f5Tc2NtLnn39OVlZWBIA4HA4JBAJavHgxSaVS0uv1fa67K1QqFWVmZtJHH31E586dIyKi5ORkcnJyIk9PT3J0dKTZs2fTunXraP/+/ZSZmUmNjY2kVCqN1ga9Xk8ZGRn01ltvka+vLwkEAuJwOMzn7msRGBhIK1eu7Jf8lJQUcnV1JS6XSytWrKCMjIxuy69fv57MzMwIAG3bto1qa2v7JLehoYFWr15NLi4u9P7779+3vMFgYD7Goq2tjWJiYsjFxYW4XC6ZmpoSn89n/g0PD6d58+aRv78/+fv708mTJ/slz2AwUGZmJjk7OzPvC1NTU1q2bBndvHmTLl26RPPnzyc+n09cLpcGDRrUr2eoO77++msCQADIy8trQJ6n5uZmWr9+Pbm5udHChQtJIpEYXUZnqNVqunLlCgUEBBCPx6Nhw4bR2bNnmXd5dXU1LVq0iKKjoykmJoaysrIoJiaGysrKeiUnJyeHXn/99XveQe3vofYPn89n/nZ1daWFCxfS999/T3FxcZSbm9una69SqWjmzJnE4XDo6aefptbWVuZYbm4uLV++nKytrYnD4ZCDgwO98cYbpFKpei2nJ9TW1tLu3bvp/fffp+rq6k7LpKWlkZOTE23ZsoVqamqM3gatVku3bt0iV1dXZpy8+yMQCEgoFJKNjQ0tWrTIKOPI2rVrSSgUUnR0NBUWFhqhF11TXl5O1tbW5OLiQqmpqV2Wu3uc1Ov1Rh0viYhiYmLI1taWLCws6Pnnn6ebN2+SRqPpsnxjYyNJJJJOy2i12k717T4p40VFRXTw4EHatWsXHT9+nD777DN6+umnydXVlQCQu7t7hx8Gl8slkUhE1tbWzPHPP//cKBfpbvbt20ejRo0iHo9Hrq6utG7dugF7EDsjMzOTRo8eTYGBgR0GCWOxe/du8vf3ZxSTuz8ODg7k5eXVYUA8ffp0v2U2NzfTkSNHyMHBgUaNGkUKhaLb8vv376fhw4cPqDJeWFhIixYtYq7D5MmTjVKvVCql1atXk1AopC+++KJHCldWVhY5OzsTAPr222/7PQE6ceIERUVFMc/O3fdYJBLRl19+ed970Bu0Wi3FxcWRn58fOTo6dvjNlJWVkcFgoKqqqgF9jjIyMmjVqlXMJBoA2dnZ0eTJk2nVqlU0b948srGxoeDgYOJwOOTi4kIeHh60ffv2PittV65cIRcXF5o1axalpKT06Jx2JeeJJ56grKysPsn917/+RaNGjeqRMp6amkp//PEHHTp0iJKSkqi+vr5PMu9GrVbThQsXGOXJ1dWVfv/9d4qLi6OzZ8/Svn376JdffqGlS5eSjY0NhYaGUnJycp/l6fV6ysrKojlz5jBjk4eHBx07doyampqYMrW1tfTzzz/ToEGDyNTUlEaPHt3vvnZGcnIyLVy4kACQSCSizz77bEDknD9/nubMmUNTpkyhvLy8AZHxV3Jycuj5558nPp9Pfn5+nRoSKisrqby8nJKTk2np0qXE4XAoMjKyV8+3Xq+n0tJS+v777+nvf/87bd++nbZv305btmyhqKgomjx5Mk2ePJk8PDxIIBDcoyC2j2X/+Mc/et3H1tZWpo5bt27do9CXlpbSF198Qf7+/sThcMjKyopWrFgxIJO7mzdv0nPPPUfjxo2jCxcudFrmyJEjZGdnR/v27etWp+oter2eqqqq6O9//3uHa2ttbU18Pv+ea87n82ncuHH9vg5SqZSCgoKIz+fTvn37mGd4oFi7di0JBAJatmxZl5NGlUpFV65coYMHD9LBgwfprbfeoitXrnSrLPeW69evU1RUFDk4ONCBAwf6VZdRlfH22Uf7R6fTkUajoZqaGoqNjaW6ujo6e/YsnTlzhs6cOUPx8fF0+/Zt+vjjj8nS0pJ8fX1p//79/erQX7l48SIFBwcTn88na2trWrZsWZ9fmH2htbWVjh8/TgKBgMaOHWv0+nNzc+m1114jBweHDpZxoVBIq1evpmPHjlF8fDxt2LCBRCIRcblcev311yknJ6dfclNTUyk6Opo8PDzo+vXr3ZZtbW2ljRs3EgCKiIgYMGV8//79NHbsWAJAzs7O9Pbbb/e7ToPBQEuXLiUej0dBQUGUl5d334FLr9fTzp07icPh0OLFi6mhoaFfbaivr6c333yTzMzMyMbGhkaPHk0REREdFHJXV1cqKSkxikVPp9PRqVOnKCIiglxcXGjTpk1GVfTvh0qlYqzEPB6PAFB4eDgtX76crl27RkqlkrRaLcXHx9OLL75I58+fp9DQUDIzMyMvLy8yNTWlioqKPllB5syZQ0KhkNauXUvFxcU9Ouf1118nCwsLGjJkCF25cqXXMtvrsLGxoWnTpjGrEH9lzZo15OfnR3Z2diQUCkkgEJBIJKLZs2f3y0qtVCrp0qVLjEV82rRplJSURBqNhhnL6+rqaMOGDeTp6UkBAQH0ww8/9MvKpNVqae/evcz95fP5tG7dOqqqqrqnbElJCS1fvpwZ21avXt1nuV3RvrI40Mp4fHw8zZ49mz2WLcEAACAASURBVEJDQ+nYsWOUlpbGfNra2owur6ioiNatW0dCoZACAwO7HXsrKipoxYoVZGlpSe7u7vT+++/36R7rdLp7PhqNhvnk5OTQTz/9RJ9//jm9+OKL5OXl1UEhd3Jy6rUyp1QqacqUKcThcKiwsLDTcbCqqoq++OILxkJuY2NDR48e7XX/7sfBgwdp/PjxNHny5C51jccff5wcHR0pNTXVqNZamUxGy5Yt6/Bcffnll3T8+HF64oknSCQSdbCOT548ucdGh64wGAz01VdfMTrW6dOnB8ToeDeLFy8mPp9P33zzTafGiJaWFjp58iR5eXkxngFcLpf4fD6dOnXKKO2Ty+W0c+dOEolEFBER0e8JSFfKeJ98xjkczj0+QiYmJnB0dMSUKVPA5/MxefLkDsdzc3NRUlICHo+HkJAQTJs2rS+iuyQ7OxuNjY3Q6XQIDAzEjBkzBiyaSWfIZDJkZWWBw+EY3a+prq4O27Ztw9GjR9HY2Ajgjg+vl5cXoqOj8d577zH+4W5ubjh69CiUSiX2798PtVqNl156CaGhob320SsvL0dMTAzS09Ph7++P0aNHd1v+woULSExMhKOjI4KCggZs53heXh4kEgm4XC4CAwPx/PPPG6VevV4Pg8HAbBTp6nrRvzfq/vDDD9ixYwcAYPPmzf2+75WVlRCLxbCxscHTTz+N999/H01NTTh69Ch27tyJsrIyVFdXY+XKldixYwc8PT37HDlHo9EgISEBe/bsQUZGBsaMGYNly5Y9kCgLRITGxkb8+OOP2LdvH9ra2pioNStXroSrqyssLS2ZDV96vR4KhQJ2dnZ47bXXsHTpUojF4g7+qr2hoqIClZWVEAqF8PHx6XHUhYiICOzbt++ezaw9RaPRQK1Ww9zcHKGhoYiMjOxwrH3jWVFRERMedNy4caitrYVYLEZSUhI8PT0RHh7ep3wCUqkUO3bsgEwmw/Tp0/Hpp58iODi4w8Y6iUSC7OxsVFdXY9KkSQgJCen3PoDz588zew/CwsKwYsWKThODeHh44K233sKVK1dQWFjIRLMxpj+qwWDo8/3rKQqFAsnJybh58yYaGxvx8ssvg8vlMv2YNWsWFixYgBkzZhgt8pVMJkNZWRn4fD58fHwgEom6LGtnZwehUAidTgcejwdLS8s+XePO2n73d8OGDYOXlxcMBgNUKhXzjvz5559x6tQpyGQy7NixA3/72996LJPL5cLPzw+XLl1ivjt58iROnz6NkSNHwtPTEwaDAWZmZpg8eTL+/PNPaDQaFBcX97p/3aFWq5Gfnw+9Xo9p06bBz8+vy3IcDgd8Pt9ov+OysjJ88MEHOHXqFAwGA0aOHIkvvvgCt2/fxjfffINbt26hqamJKW9ra4tRo0YZxV88Ly8Per0e7u7uGDly5IDHdW9/Vru6dlKpFHv37kV5eTkAYMOGDQgJCcGaNWuwdetWBAQE9HuDfG5uLuLj42FqaoqIiIgBi+Zi1NCGHA6H2dR09+YmhUKBkydPIj4+Hj4+PnjyySeNuqli/fr1zMaF0NBQvPjii5gyZcoDjTXa3NyM0tJSWFlZ4ZlnnjFq3RqNBhKJBI2NjTAYDBCJRJgyZQreffddeHp6wsHBgVEcvb298eyzz2Lbtm2oq6vD8ePH4ezsDB8fH9jZ2fVKbllZGZKSksDn8zFnzpxud8M3NjYiMTER2dnZ8PPzw7x58wYkZFdRURFu3ryJ2tpa8Pl8ODs7G33SlZKSgvfffx+Ojo6IiIjokMimtbUVmZmZSE1Nxfnz51FfX4+pU6fCzs6u34OtWq2GVquFv78/Zs6cCUdHR9jZ2eGZZ55BXFwcysrKQERoa2vrdELcG3JycnDgwAEkJydj0qRJWL9+PRwcHPrV/p7S1taGpKQk7Ny5EwqFAg4ODtiyZQsmT54MFxcX5ndD/97sKBQK4ejoiLfffpuJfqHT6RAdHQ0bG5teX4fTp0+jrKwMgYGBCA0N7fEkqj1qQ1+5fPky8vLy4OrqiuHDh8PU1BQ6nQ7V1dU4ceIEdu7ciaKiItjY2OCJJ57AggULmA3x+/fvx6FDh1BVVYXGxsZeK+OFhYX49ttvERcXBysrK7zzzjsIDAzs8Izq9XrExsYiOzsb9vb2GDt2LEaOHNnn/rZPutLS0jooooMGDep0omtiYgInJyfMnDkThYWF/5XJPRQKBb777jvs378fBoMBgYGB8PLygq+vL3Q6HU6dOoXk5GRcvnwZtra2+PrrrzFy5EhYWlr2K/pVc3MzpFIp7O3tsWjRom5/pwUFBRCLxTAYDBg8eDBCQ0P7LLc7TExMGGXNwsICIpEIdnZ20Gg0OHXqFMzNzREVFdWrOtsNMMCddwGXy8WWLVuQk5ODM2fOMJv4dTodkw9Bp9Ph6NGjCAkJwbhx44xiJJJKpSgrK4OpqSmGDRvW6btOoVCgtLQUQUFBRosCUlVVhYMHDyImJgZtbW3w8PDASy+9hL///e+or6+HTCZDW1sbo8RaW1sjKioKL774Yr/1ISLC5cuXQUSYN29en8ZeY9Lc3IyUlBScO3cOAoEAH330EZYuXQpLS0uMHj0aly5dYsLw9ge1Wg2FQgGBQAAvL68B6/MD0VZv3ryJpKQk1NfXY/To0fDz8zOaRUAulyM2NhbFxcXg8XiIjo5GVFTUA02XKpfLkZaWhrS0NPj4+Bg9dBX9O8QaEcHa2hqTJk3C6tWrERISck9EB1NTU0yfPh3nz5/HtWvX0NDQAJlM1msrolarRWVlJfLz82Fra4t58+Z1W/7mzZtIS0uDRqOBv7//gA3whYWFkEql0Gg0GD58OEJCQowSRYXD4SA6OhqxsbEdQgleuXKlw0xYq9Wirq4ODQ0NaGhogJOTE1avXm2UiUdeXh4qKyvB5/OZFRAejwc7OztUV1eDw+GAiODr6wsrK6t+x7ouKCgAl8tFUFAQxowZ88CSjuj1esjlckaxbo9EcPXqVdTU1ECv14PH46G5uRkVFRVMXNfc3Fwm6oeNjQ1WrFjRp5fc6dOnYTAYMHv2bLi7uw9IXP7OyMvLQ21tLTw9PZm49TKZDL/99hv27t2L/Px8jBw5EitWrEBERASTJITD4SAgIACDBg2CVqvt04pAfX09rl27hpaWFri5uSEsLKzDc2MwGHD27FmcOXMGVVVVmDhxIqZOndovy5dOp0NhYSHKysoAAH5+fpg2bVq3dZqYmMDOzo6ZdMrl8ocWQaEv1NTU4MaNG9Dr9Vi8eDHmzp0La2triEQiGAwGTJs2DfX19Th79ixSU1OxceNG+Pj4YMmSJRgzZkyffs96vZ7J72Bra4thw4Z1W/bMmTPIysoCADg4ONw3ApqxMDExARFBKpUCuKPkFBcX9yofhYmJCaKiovDqq69i6NChMBgMkEqlkMvlaGho6PQcnU6HnJwcbNy4Ee7u7li5ciVCQ0Nhbm7eZ8WqrKwMEokE9vb2XWbnvXr1KqRSKWbPnm2UFUciQmlpKY4cOcJYvhsbG3Ho0CFkZWVBKBQiKioKVVVVKCoqgkKhwJAhQzBlypQuLfe9RS6Xg4ggFotx9OhRNDc3Iz8/H3K5HMCdKCNeXl6YOXMmxo0bN6DKulgsRlxcHIgIa9as6ZAPxMLCAkqlEunp6XB3d+9XKNra2lqUlpYyBoqBYsCVcalUipMnTyIrKwv+/v547LHHjJoy+5dffkFJSQk0Gg2CgoIYS+aDzGZWU1ODtLQ0SCQSzJw5E4MHDx4QOXq9HqGhoVi1ahXCw8O7DK3m5OQEOzs78Hg8EBHKysqQm5sLR0fHHsvS6XTMLDssLAxeXl7dli8sLER1dTWsra3h6ek5YC/Q4uJitLS0gM/nIzAwEBEREUarOyoqCu+88w4KCgpQWlqK1NRUNDQ0MKsPwJ2H3NvbG7/++is4HA78/f2NlvXU3NwcAoEAEokEt2/fZpTQrKwsVFdXQygUQqlUory8HA0NDRCJRH22drQr/kqlEhUVFUhPT8fQoUMfiHW83cXK29sbeXl5aG5uxq+//gqFQoGGhgYQEbhcLtra2iCTyaBSqZiY5jweD4GBgViwYAEiIyP71P+Kigo4Ojpi8uTJ/Rqke4tUKkVbWxsEAgEEAgE0Gg1KSkpw7NgxFBUVYdy4cVi1ahWio6Ph4ODQ4UUmFAphb28PFxeXPhkatFotmpubmSXzu11zmpqakJaWhj179iArKwsGgwHDhg3rl1W8Xebt27cZ69RTTz0FX1/fHt+z2tpalJeXGy1E618RCoVGj2duZWWF+fPnY8aMGQgLC7vHNcDX1xdKpRJDhw5FQUEBTp8+jdTUVKhUKrS2tvYpjKTBYEBzczPq6uowaNCgLjMFGwwGVFZW4vr16xCLxXBxcUFQUNB9E5oZC61Wi+LiYiY8HI/H69U7CbhjNBk6dChefvlleHp6oq2tDY8//jiSkpIgk8lQU1PDKIZ3o1AokJiYCD6fD6VSiQkTJmDu3Lnw8vLqdYhSg8GAtLQ0iMViBAQEoKGhAcePH2dyIlRUVAC4EwKwtbUVIpHIaBN+tVrdITZ+W1sb0tPTMWHCBERFRcHX1xc7d+6EXq+HSCTC+PHjMXHixPtmj+4tV69eRV5eHlpaWpj+AndW8V1cXJhV3P7mWmm3visUCsbVrV1OYWEhkpOT4efnh2XLlsHd3f2e82tqaqBUKvs8zuv1eshkMsjlcowaNQoikQi1tbXM8XY9yxgMuDKelZWFxMREtLa2IioqCnPmzOnWn603VFVVYefOnWhuboa9vT3mzJmD4cOHP9DkKMAdn+6ioiLw+XyMHDnS6JY2sViMlpYWAMCIESMQHR3dbXmJRIK6ujqo1WrGwtR+fm/gcrmwtLSEg4MDtFptl9ZfjUaDoqIi1NfXw83NzaiTrbuRyWRIS0tDQ0MDhEIhhg4dalQXFRcXF7z++uuoqKhATk4OLl68CLVajWHDhjF9Mjc3h5mZGX799VfY29tj7ty5vXb/6Yr2hAG5ublITEyEmZkZKisrcevWLYhEIkyaNAkxMTG4efMmrl69Cmdn5z4PMqWlpZBKpUxKeB6Ph9GjRyMyMhL29vZQKBSwtrZmBsOWlhZYWFjA0tKy34OPQCBAQEAAnn/+eaSnp+PWrVtIS0uDQCBgrFVyuRyNjY1QqVQAwCSIGT58OF544QUsW7asT+NI+74Ae3t7uLu7D3iSsrtpbm7u4Hohk8lw7do1FBQUYOjQoVi5ciWeffbZTseP9ljGrq6uffIX5/F4MDc3Z8aDM2fOMMeqqqpw9uxZnDt3Ds3NzXB3d4e3t3e/Vxf1ej3EYjGzZL5w4UJYWVn1SLGmf+c2aE8sMxDKuJmZmdGVcScnJzz77LP3lRsWFoawsDAMHToUFy5cwKlTp3DgwAHo9XqMHz++V2NK+wTL1NQUWq0WtbW18Pf371BGpVKhtLQUp06dQm5uLjOuRUREPLBsnE1NTUhPT0dycjJ4PB4CAgIQHh7e63oEAgHjqmJqaoqlS5ciKCgI9fX1KCkpgVQqhU6ng06nA4fDgUAgABEhKysLxcXFOHHiBFJSUgCgTwq5VqtFVlYWKisrYWFhgYMHD6KoqAjAnXvR/rdEIoFOp8OtW7dQWFgIS0vLfq00cTgcDB48GHPmzMHNmzfB4/Hg4OAAe3t7TJ8+HVOnTsWlS5eY/kdERGDmzJldWu77046KigooFAq4u7sjNDSU8dXPz89HeXk5jh8/Dh6PB1dX106V5J4yduxY7N27F9evX8fChQuZyZtYLEZycjLq6urw6KOP3mP5b2trg1AohKura7+uefskV6VSobGxEceOHetwfPjw4QgKCoKrq2u/V8cHVBknIib5wfDhwzF+/Hi4uLgYpW6tVotTp06htLQUer0egYGBmDt37oApgl3RvkwmFovh6OjYYVOWsUhISIBYLIaJiUm3LyWdTgelUolz584hLy8PKpWKsSCHhYX1SqapqSmcnJwgEomQm5uLjIwMDBkypEMZpVLJzNRTUlIglUoRHBxstHv8VwoKCnDt2jXIZDK4urrC0dHR6JtEzc3NMXz4cAwfPhzz58+/53hLSwtiYmLA5/MxceJEvPDCC0aT7enpieDgYFy9ehUJCQm4ePEieDweBg8ejPnz52Pt2rUoKytDSkoKTp48iUmTJvXZXcXMzAx8Ph8ajQZisRi//vor/vjjDzz22GMYMWIEampq4ObmBm9vb3A4HFRVVcHNzQ0+Pj7w8vLql6LWvsn5nXfeQUtLC/bt24fMzExYWVnB0dGRsTwlJiaiqqoKwJ0VCV9fXyxduhQvvfRSn+97dXU1Wltb+6SAtC/R9lUxbE8Q0q6c1tTU4PLly4wCtnTp0k7P0+l0qKurY6zqfbnf9vb2GDduHEpLS1FXV4c333yzQ79EIhGEQiGTXKPdp70/GAwGJuEJcOfF9SBXLP8bmDBhAoKCgqBSqfDnn3+iubkZKpUKUVFRPVbIuVwu3NzcEBQUhIqKCsTFxcHZ2Rm2trZQKpXQ6/UoLy9HXFwc9u7di5qaGvB4PCZ51YNAr9ejrKwMly9fhlqthqWlJRYuXGgUX2o/Pz94eXmBz+dDr9dDp9MxCaTaE5QZDAbs3r0bv//+O/Ly8lBdXY19+/aBz+dj/vz58PDw6FVflEol2trakJubi4aGBjg7O8PLywsODg5MoIPdu3ejubkZMTEx8PT0hLOz8z3v0N4ybNgwfPDBB8jIyGDuYXviNbFYjP3796Ompga2traYM2cOoqKijLp3y8HBAVKpFIMHD8ajjz6KadOmwcXFBUOGDAER4fr16/jjjz9w+PBhHDp0CI888ggWL17cZ3lz5szB+vXrceLECSxfvhxubm4QCoXIy8tDQkICrK2t73EdkclkqKyshI+PD8aOHduv1c/GxkZm70FJSQm2bdsG4M4KRWtrKzw8PPDMM89g1qxZGDFiRL8U/wFVxmUyGW7cuAGlUonp06cjIiLCKINxu8/Szp07oVarIRQKERwcDDs7uwc+2Dc2NqKgoAD19fWYPHlyr5XenpCRkYG6urr7lquoqEBGRgZ++eUXyGQy8Hg8ODk5wdHRsdcPpImJCRwcHODg4IDr16/j3XffveehKi4uRkVFBaqqqnD79m3GQmPsJbF26urqoNVqweFwMGzYsAfm69hO+wvll19+gZmZGV5//XWjrfIAdxTkBQsWMJkem5ubIRKJsGLFCqxduxYCgQBz585FTk4OUlJSkJWVBWdn5x5HA7mbadOmoaSkBMnJyUy6e4VCgUOHDnV5jrW1Nfz8/LBixQo8++yzRrnPVlZWeOWVVzp819bWhq+//hppaWkA7ijiISEhWLt2LR577LF+PePnzp2DWCzutXVZq9Xi/PnzUKlUsLa27pOi2q7E363MtyvmEomEWX34a//EYjGuXLmCmpqaPrsR+fr64u2334ZcLsfFixeZpdb2dPRr165FYmIi4uPjERISgqCgoD7JuRutVov8/Pw+TV44HE6fszT+t2FpaYnnnnsOwJ39DF9//TUUCkWPo0RxuVxMnToVKpUKb7zxBr777jskJyfj0UcfRWFhIVQqFbKystDS0sK4etnZ2cHNzc2o41dXEBFqa2tx8eLFDhbT++1D6glqtRoVFRUoLS3FsGHD4O7uzoxLf53MvP3225g8eTLWr1+PK1euIDs7G7/88gu8vb0xePDgHq/4tUesGTFiBPz8/LBkyRJERkZ2UPoMBgMOHz4MtVoNJycnZGRkQCaT9VsZNzExgbOz8z3RiIgIMTExuHjxIlpbWzFt2rR+K6J/hcPhYPbs2cjNzUV4eDg2bdp0j1Fm7ty5cHFxQUpKCsRiMWpqavol09HREePHj8eZM2dw7NgxODg4YOTIkVCpVNBoNBAIBB2uhVKpxLFjx5CVlYWNGzf2e9Xa1tYWo0ePxty5cyESiTB48GAYDAZIJBJkZWUhJycH//znP5GUlIRNmzb1OTMzgL5l4OwJer2eNm/eTC4uLjRu3DiKiYnpV313o1ar6d1332ViL8+bN4/KysoGJJva/YiPj6dZs2aRl5cXffLJJwMiY/HixWRjY0M8Ho/eeOONTsuo1Wpas2YNCYVCJt6mi4sLffLJJ1ReXt4nuY2NjXTy5EmaPXs22djYkFAo7PBxd3cnT09PsrS0ZOJ7rl271qjB9u9mzZo1ZGdnR5aWlvThhx+SWCweEDldUVlZSZ988glxuVxyd3enW7duDZicvXv30urVq2nTpk0dYqUqlUpasmQJmZmZUXR0NKWnp/dJhsFgoMrKSvriiy8oICCAvL29mZjWf82k99dPZGQkHTt2zFjd7YBer6effvqJAgMDmXjQ4eHhFB8fb5T6//WvfxGPx6PQ0NBOY113hsFgoOvXrzMZJLdv3051dXW9lr1q1SpycHCgxx9/nC5cuEAajYYSEhLIzc2NRCIRffzxxx2yNer1etJoNLRmzRry8PCgVatW9TgmeleoVCq6ePEibd26lbZu3UpnzpwhnU5H69atI0dHR3J3d6cff/zRKPF56+rq6JFHHmHihmu12vueI5PJaMOGDcThcMjOzo7S0tKMGp/5rxk4e9KmB4VCoaCtW7eSu7s7TZs2rdfnS6VS2r17N40bN44Zo62srCgoKIhWrFhBp06dopEjRxKXy6WJEycOSOztzmhubqaff/6ZxowZQwDI0dGRdu3aZZS6z5w5Q6GhoWRlZUXvvvsu5efn3/f30tLSQpGRkSQUCsnCwoLefvttys/P75VcpVJJcrm8y+ekoqKCrK2tadSoUZSQkEAFBQUDGpO7pqaGxo4dS6ampuTq6kqxsbFG14cMBgN9+eWXTC6OrsbP/Px8evLJJ8nS0pK++uqrfss9ceIEk1AyLCyMtm7dSq+++ioBoFGjRpFYLCadTkcqlYr27dtH7u7u5O3tTbdu3RowXYTozm8gLS2N5s+fz+R76UlGUqPGGe8J586dw+bNm6FQKLB48WKMGTPGaHVrtVpmuQAA/vnPfz7wTZvtNDY2QiaT3bMp6kGi1+vxxBNPMDFA22n35+rNEtzdiEQizJkzB9OmTUN2djYTy7OdJ554AjweDx999BH27NkDsVgMHo83ICEN28O/yWQyeHp6YsSIEXB1dTW6nO6QSqW4du0azM3NMXPmTAQHBw+IHDc3Nzz33HOMtexuhEIhnnjiCcTFxSEtLQ3l5eXw9/fv9T6J9pj0a9euxdq1a9HS0oIbN25AoVBg06ZNjE9lZ9TU1DARMozN4cOHsWXLFuTn54OIMGjQIIwfP35A3L96AhEhLS0N27ZtQ319PcaMGYNHHnmk1246tbW1kEgkjA88cMfCFh4ejj///BNLly7Fxx9/jPj4eJw8eRI2Nja4cOECvvvuOyQmJmLixIlYtGhRvy1rAoEAUVFR94SUO3DgAOrq6hAVFQVvb2+jhWLr7ZisUChw+PBh8Hg8rF+//r65DXqLq6srAgICcPv2bcjlcmzevBnr1q0zqoy+YmFhwaz4GQyGXp/v6OiIZcuW4emnn2aipZiamjLv3tLSUiYiU3BwMAICAoza/q44f/489uzZg7S0NAiFQgQEBGD58uVGqVuhUEAqlaK1tRVbtmyBVqvFm2++2W3AAUtLS2zbtg1PP/008vPzmRjl3UWh+StCobDLlUEiwtmzZ9HS0oLp06dj1KhRA64bPPnkk0hPT4dOp8OyZcvg7+8/IPqQubk59Ho9NBoNNBrNPW57crkcN27cwIULF4wWz//xxx9HXV0ddu3ahYyMDFy/fp051traisTERGg0Ghw/fpzx6b5w4QJGjBgxILpIO0KhEE5OThCLxcyG2f649g2IMi6TyfDee++hoaEBU6ZMwaRJkwZ0x3ZjYyMaGho6bHwSCAQwMzMDl8tl/MeICHK5HN988w3zQzExMcFLL72EgICAPm28zMnJQW5uLoKCgjBhwgSj9emv0L/DG5aVlSE2NhZlZWX49NNPmeXmdveNu1m/fj1mzJjRb9lmZmYYO3bsPeEK2+UNHjyYeXnX1dVBLBYb3Xe/3d0JuLOpo7+B/PuCRCJBSkoKnJ2d73GteJDMmzcPFy5cwO+//46jR4/C3d2935NdKysrPPLII0zymZSUFPB4PCxZsgTPP/88Tp48iSNHjjA+3ANBWloaNm7ciOLiYuh0OggEAsyaNQsvv/yy0QZVLy+vXiU50Wq12LJlCw4ePAgnJycmPFpvXS8cHR0xZMgQXL9+HYWFhcjNzUVERARMTU0RHByMa9euIS4uDv7+/mhpacH69euxd+9eaDQaLFy4EK+//vqAhQutrKyEXq+HhYUF5s2bZxQXFeBOJISNGzciMzMT9fX1ePnll/H9999361ep1+vR2NgIR0dHrFy50ijt+G+hpqYGV69eRXFxMaZOndqnOjgcDiwsLDqNYiGRSJjQoF5eXv3aWNdTrl69il27duHy5cvg8XgYNWoUvv/+e6NtyF2wYAFkMhk+++wzlJeXg4h6pATm5uYy7xNXV1ej6yftoWknTZo04AElSktLkZ2dDZ1OhylTpuCZZ54ZkHvL4XCwevVq3Lx5EwcOHMCHH36I9evXw9fXF1wuFy0tLYwRVi6Xw9LSsk+bzTtj2bJleOyxx3D8+HHs3r0b6enpAO5EcXv66aeZcuPHj8dvv/0GT0/PATXOEhGuXLmCN998ExkZGQgJCcHChQv7FUnP6Mq4wWDAwoULkZubCxcXF6xevbpXM86+MHPmzHse7mHDhmHMmDGwtLSEWCxGbGwsgDsX8W7rFHDnhfvKK6/0ejNLQUEBCgoKGAf/gYqp+dhjjyE3NxeZmZmIi4vDpUuXQETM5px27pY/d+5ceHp6GrVN96uLiGBvbz8goR3bs6va2Njg8ccfZ3bTPyhycnIQFxcHuVwOT0/PBy7/bkxMTLB8+XIkJibiyJEjCAgIgJeXV7/9twxVpgAAFuVJREFU49p9/qdOnYoNGzZAp9Ph999/R1VVFePzCPxvAg9jEx8fz0QCEAgEWL58OVatWmXU8SMyMhKurq5QKBSQyWRwdHS8x1dUqVQiIyMDv//+OzIzM5GUlASDwYDvvvsOkydP7rPV+K233kJraytOnDiBLVu2IDs7m4nGo9PpkJaWhu+++w6pqaloa2vD4MGDsXr1aixYsKDPq1s94euvv0ZzczPGjx+PsLAwo71A+Xw+QkND8Y9//AOvvPIKjh8/jhUrViA0NLTTyVVpaSk+/vhjyOVyPP7440bP7ldaWorExETk5uYatd6/cu7cOUgkEgQGBvbYsl9fX4/Dhw8zPvsDYa2Pj49nwv71N2lYT8jMzMTmzZuZRDEhISHYsGGD0WJeA3f68eSTTzKb2g8fPozCwkLMnDkTCxYsuGf1dNeuXfjtt9+Ql5fHjGd3Z0c1Nv1NFNYdRITq6mrMnTsXTU1NcHFxwUcffTSgiWk4HA4+/fRTJCYm4sSJE0y+Bnt7e8THx+PgwYOQyWTw9fXFokWLsGjRIqPJdXZ2xksvvYRnn30WP//8Mz766CM0NjbCysoKL7zwAsaMGYP58+f3O4FWZ7QbQ01MTNDU1IStW7di586daGhoQHBwMDMp6U+kMaMr4xUVFbhx4wY0Gg1eeeUVhIWFGT3ihYmJCSIjI3H+/HkA6JD6tZ2MjAzk5+eDy+VCp9OhpaUFPB6PuUmRkZEYMWIETExMMG7cuD5Fh8jIyEBxcTFEIhECAwPvCSdlLP5fe/ceFOV1PnD8u7vgKjdhQW7LRe5iFoaLCKiItyAipFhjFBPbkI4mjXRiGtQhzYzTaWPasU5majvOZMbJ1E4qauhY0wQSotJwUxoDwbsGo6AEiRXEXS7usvv7g9/uaNS0xj37puV8ZjKJToZn35f3nPfZc3tmzZrFe++95yiTPTIy4hjFV6vVWK1WVCoV/v7+REVFUV5eTlJSkmOXtWh3Nny1Wi3kG2lzczO3b98mODiYwMBAYZtEH+TMmTPU1dXh4+NDZmYmGo0Gk8nEwMCAsNNjvs1jjz1GaWkpO3bsoLq6mvj4eIqKih752EF7uen8/HxqamoYGRmhqamJkZERrFYrCQkJjt3jzjQ4OMiuXbswGo3A2OzH4sWLSUhIcOrzZO8DLl68yMaNGwkLC7snMTQajbS3t3P58mWGh4eZNGkSBQUFZGdnP9LyjdDQUF555RW0Wi379++nsrKS6upq1Gq148hBk8nE4OAgRUVFvPTSSxgMBvz8/IQVJuro6OD9999naGiIjIwMgoKCnPoit5eQXrp0KTU1NbS0tNxT+RPGZlPr6+s5cuQIwcHBrF+/3mmfwa6+vp66urrvtATkP/Xpp5+yc+dOhoeH8fb2fmAy3tfXR11dHRaLhZaWFtrb27lx4wYGg4FVq1Y5febv2rVrfPDBB/T39xMXF0dYWJjQYz1NJhO/+tWvHFUQp02bxooVK5gzZ47T3w++vr68+uqrmEwmamtraWpqorW1lZ07d97znrh27Rp9fX2OJRaRkZEsW7ZMaDEXUaxWKydPnnQs6SsuLv5OSxYfVnBwMNu3b2fTpk3U1tbS0NCARqPBaDTi5ubG0qVLee655+5blPBR2I+p1Gq1TJ8+ndTUVA4fPozVauX69es0NTWRmprq1OWjZrOZS5cuOWp99Pb20tDQwIkTJzAajaxdu5a1a9cSFxf3yEeEOi0Zt/3/junNmzdjMplYs2YNxcXF9xSvcIYJEyawdetWdu7ceVc1up6eHtra2rh+/TojIyP4+vo6yl27ubmRlZXlmL4JDAxk8uTJqNVqvL29H3rXsdlspr29nStXrhAREUFqaqrT1ll+k16v55lnniEoKIhPPvmE9vb2+/5/JSUlrFixgqSkJDw9PZ12GP2/YzabsdlsuLu7C0kaRkdH+eKLLzCbzY61eq6qmmg3NDTkOCrLzc2N2tpaDh48SEhICBUVFS4vC6zValmyZAl1dXUcO3aMY8eOMXv27IcuonE/kydPpry8nBs3bnDhwgX6+/vR6XSkpaWxcuVKHn/8caeNnsLYGcgvvPACly9fxmKxEBISwtNPP01WVpaQNX/2kZVjx45x/Pjxe5IDs9mM0WjEYrHg4+PDihUrWL9+/SP3ZRqNhujoaNatW8e0adNobW3l4sWLtLe3k5OT4/j/8vLyyMzMJCEhwVGdVJTTp087qp76+Pg4/X6r1WpiYmIoKyvj+PHj7N69G41Gw6pVqwgICODmzZt0dXXR0NDAn/70J27dusXy5cuduscIxgaJmpubOXfuHDC2n6a8vFzIl8re3l5OnTpFX18fR44cIT4+ntHRUXp6evD29ubo0aOMjIzQ29uL1WpFo9Gg1+spKSkhNzeX2NhYp/8eurq66O7uZmRkhOTkZCIiIoT1ofYp/JMnTzIwMMCUKVMoKCjgySefdPrAHIw9Y5GRkVRUVDBr1iwaGhr4/PPP6enpccxa3/nZYOx0IYPBwJIlS1i4cKHTakXYY5w+fdoxcn3n7LWz2KuK/uY3v+H27du4ubmxYMGC//gc/0ehVquZO3cur732Gn/5y19obm5m0qRJpKWlkZOTw9KlS4mLixPyu7ZLS0vjhRdeAMaWqlitVkJCQpyWg9lsNlpaWjhw4ACnT59meHiYrq4uTCYTfX196PV6tmzZQl5eHlFRUU5pr07L1kZHRzl06BA1NTXYbDaWLFlCRESEkIRQrVaTkZGBSqW6a5Sjv7/fcRg9jB3FFhUV5TguLCYmxqmVIQcHB9HpdOTm5jq1EuQ3abVaZs+eTUREBElJSdTW1tLW1kZXVxeJiYlkZmZitVopLi4mOzvb5YnqiRMnMJlMpKSkCJsd8Pf3R61Wo9PpXD4qfiej0UhdXR3nzp1z/F5cnYjbTZ06lTVr1nDt2jU+++wzWltbnbJHQKPRkJWVRUVFBTU1NZw9e9ZxbGdaWprT11eePXuWgwcPMjIyAsCSJUuYM2eOUxP+Oy1fvhw/Pz/efPNNOjo67irEA2Pr5zMzMwkLCyM3N5e0tDSnjba4ubmRmJjIlClTyMzM5Pr163R2dt41dW8wGJyaHHyb7u5ubt++TWBgIAaDQUhFUg8PD7Kzs9m8eTN79uxh7969XLt2jczMTC5dukRLSwtnz56lo6ODpKQkiouLnf4it1d3tVdeXbhwIXl5efccEfeoYmJiWL58Of7+/vT39zv2XwwNDeHu7o6/vz8DAwN4eXkxb948VCoV0dHRREdHYzAYHuqIvYdhLwJnr48g8ljYzs5Odu3aRXd3t2Oz6Pz584mIiBDWV6rVapKSkggMDCQlJYVLly5x9epVRwXjqKgourq6mDJlCmFhYSQkJBAdHU1CQoLTj0S22WyOI/1EzMLY977t27ePo0ePAmObUsPCwlz27vfx8SE/P5+goCCeeOIJJkyYgF6vJzo6mqioKOGHaeh0OubPn4+Pjw/d3d0EBQURHBzstFnqxsZGdu/eTU1NDdevX0er1WI2mwkLC6OoqIicnBwWLFjg1IrVTmn1VquVK1euUFlZ6fgmGhoaKvzBELWh6T+h0WjIyckhIiKCtLQ04Wde63Q6dDod4eHhJCQk8Omnn9LZ2YnBYGDWrFlYrVaX3PP76evrIykpiYKCAiHnrKvVavLz8+ns7CQ+Pt5lFePupNfrSUlJoa2tjYkTJxISEsK8efOckvx+V1qtlsWLF9PQ0MCHH35IQ0MDs2fPdkoi4+HhwQ9+8AP0ej1ffvkl2dnZhISEOP35stlsVFVVOfqNxMREli1bJnTdY0xMDFOmTGFwcPC+ybiPjw/p6emEh4cL2ZSt0WgICgpyWRnyb/P5559js9lITU1l+vTpwtqWj48PpaWluLu7889//pPGxkZOnTrF8PAwN27ccJyhP2/ePCF9iJ+fH4sWLSI9Pd3xbxH0ej1PPfUUSUlJ9Pb2cvPmTS5cuOD4wuPl5UVwcDBeXl4kJyej0Wjw9/cXPgNy5swZbt++jU6nY/r06U5NIu5kNpv585//zMcff+yokzB79mxSUlJcMmhhb1dZWVkYjUYGBgb47LPPiImJ4fLlywQHBxMeHo6fn5+wZTr29c1Tp0595HXE9zM0NMTx48epqqpyLKMrLi4W0j9/G51Ox8KFC7/zZuNHFRAQIOz9e/XqVbq6uoCxQa/4+HgiIyOJiYkhLy/PqZW/7ZzylIyMjNDS0kJtbS0wdvrGv6sW+d9OrVZTXFzs8rg6nY65c+cqdtTb/eTm5hIZGcnMmTOFjGaqVCqWLVuGxWLBz89P2Ijpt0lOTuYnP/kJLS0tJCcns2jRIqeUhn9UXl5ezJ49m/b2ds6dO0d3d7dTNzzOmDFD6Jdem83Gu+++i81mY8KECZSUlDBz5kyhU5wwlhy+9NJLQmP8Nzh58iRWq5W0tDThX3J9fHx49tlnycvL4+9//zvNzc3ExcURGxvrqFwYHh4uZFQtJCSEdevWOf3n3o9er3f5sav/zscff8zw8DDTpk3D29tbWJzu7m7++Mc/cvPmTby8vMjJySE3N9fl98PNzQ1fX198fX0dm59dtelepVJRUFBAeHg4KSkpTl9yZDQaaW5u5uLFi0RHR5OUlMSrr74qdOZhvHnsscf44Q9/SEpKCiEhIcyfP5/ExESh73un/GSz2exYV6PVasnMzESn08kHY5z46U9/KjyGTqdT9Kgzf39/CgsLKSwsVOwzPEhRUREDAwO0tbVhMpmU/jgPzd7BRUZGUlxc7LIlGtLYTER7ezsTJ050SX/t4eFBbGwsGzZsYMOGDcLjSWO++uorIiIiyMjIENq+Tp06xeDgIFqtlqysLH72s5+RlZU1rnIB+wkvTz75pJCfr9FoCAgIIDU1lWeeeYaVK1cKm+kYrwwGw0OfrveonJKM23e5Tp48GYPBwLvvvvudTieRJOnhabVannvuOaU/xneiUql4/vnnee211ygrKyMqKkrx2YbxZMGCBVy+fJmUlBRhG9Al5dnPZRYtPj6e6OhoAgIC2LJlCzNmzFB0j8//In9/f8rKyigrK1P6o0hOpOrv73/gCfkiNvNIkiRJkiRJ0nhjsVjuO4P9rUNQVqtV6JmskiRJkiRJkjQePGhPzLcm47du3RLyYSRJkiRJkiRJArGHQUqSJEmSJEmS9EAyGZckSZIkSZIkhchkXJIkSZIkSZIUIpNxSZIkSZIkSVKITMYlSZIkSZIkSSEyGZckSZIkSZIkhTg9Gdfr9Xf9o9Pp2Lhxo7PD3GNkZISysjIMBgNhYWHMmTOH2tpa4XEB3nrrLebNm0dgYKBLSsPfqa+vj6effprQ0FAMBgP79+93afyOjg6CgoJYt26d8FhK3mc7V14vwLlz5ygqKiIiIoLU1FTee+89l8QFWLp0KUFBQY62PGPGDJfEVeqalWxLVVVVzJw5k9DQUFJSUmhqahIeU8lnS6l7rVQfMt7aklJ5AMC6detISEggPDyc9PR0du/e7ZK4duPpHQHK9F1KxRXZjp1ed/rq1auO/zYajSQkJFBcXOzsMPewWCzo9Xref/99wsPD+eijjygtLaWxsZHIyEihsYODgykvL+fw4cMMDQ0JjfVN5eXlTJgwgfPnz3PixAlWrlyJwWAgMTHRZfHT0tJcEkvJ+2znyuu1WCysXr2a0tJSDhw4QENDAyUlJSQmJhIbG+uSz7Bt2zZ+9KMfuSQWKHvNSrWlI0eOsGXLFt5++23S09Pp6ekRGg+Uf7aUutdK9iHjqS0plQcAvPzyy+zYsQOtVsv58+cpLCwkOTmZlJQUl8QfT+8IJfouJeOCuHYsdJnKwYMHCQgIYNasWSLDAODp6UlFRQWRkZGo1Wry8/OJiIigra1NeOwnnniCwsJCdDqd8Fh3MplMHDx4kF/84hd4eXmRnZ1Nfn4+e/fudUn8qqoqJk+ezNy5c10ST6n7bOfq6z1//jw9PT2sX78ejUZDbm4umZmZVFZWuiS+EpS6ZiXb0htvvMGmTZvIyMhArVYTGhpKaGio0JhKPltK3mul+xBX+r70H67MAwASExPRarUAqFQqVCoVX375pUtij7d3hBJ9l5JxRRKajO/Zs4dVq1ahUqlEhrmv3t5eOjo6XDZCrIQvvvgCNze3u74BJyUlcebMGeGxBwYG2Lp1K6+//rrwWN8H35frtdlsLvn92v3yl78kOjqaxYsXU19f77K4d3LFNSvVlkZHR2ltbeVf//oXqampTJ8+nY0bNyoy8+OqZ0vJfktJ46UtfZMSecArr7xCSEgIGRkZBAUF8fjjjwuPOd7eEUr1XUr3maLasbBkvLOzk8bGRkpKSkSFeCCz2czatWspKSkhPj7e5fFdxWQy4e3tfdff+fj4YDQahcd+/fXXWbNmDXq9Xnis7wMlrjcuLo6AgAB+//vfYzabOXz4MI2NjS7tdNra2jhz5gw//vGPKSkpET7CpNQ1K9WWent7MZvN/O1vf6O6upr6+nra29v53e9+JzSuks+Wkv2WUsZTW7qTUnnA9u3buXLlCtXV1RQVFTlGykUab+8IpfoupeKC2HYsLBnfu3cvWVlZTJ06VVSI+7JarTz//PNMmDCBbdu2uTS2q3l6enLr1q27/m5gYAAvLy+hcdvb2/nHP/7Biy++KDTO94VS1+vu7s4777zDhx9+SHx8PH/4wx9YtmyZy6bjZsyYgbe3N1qtltWrV5OZmclHH30kNKZS16xUW5o0aRIwtuksODgYf39/Xnzxxf/Z+wzK3Wsljae2dCel8gAAjUZDdnY23d3d7Nq1S2is8fiOUKrvUiouiG3HTt/AaVdZWcmGDRtE/fj7stlslJWV0dvby/79+3F3d3dpfFeLjY3FYrHQ0dFBTEwMACdPnhS+NKehoYHOzk4MBgMwNtI1OjrK2bNn+eSTT4TGVoKS12swGPjggw8cf87Ly1NktgnG1l/abDbhcZS4ZqXakq+vL3q9/q4pfFdN5yv1bCl1r79P/pfb0p2UyAO+yWKxCJ+FGI/vCKX6LiX7zG9yZjsWMjJ+7NgxvvrqK5ftnrb7+c9/zvnz56msrHR8e3IFi8XC8PAwo6OjjI6OMjw8jMViER7X09OToqIitm7dislk4ujRo1RXV7Ny5UqhcZ999llaW1upr6+nvr6e0tJS8vLy+Otf/yo0rlL3WanrhbEkZXh4mMHBQXbs2EFPTw+rV68WHre/v59Dhw457vG+fftoampi0aJFwmMrcc1KtSWA1atX89Zbb/H111/T39/Pzp07Wbx4sfC4Sj1bSt5rJfqQ8daW7JTIA77++muqqqowGo2Mjo5y6NAhqqqqyM3NFRp3PL4jQLm+S4m4otuxkJHxPXv2UFhYeM+6QJE6Ozt5++230Wq1JCQkOP7+zTff5KmnnhIae9u2bfz2t791/Hnfvn1s3ryZiooKoXFhbG3c+vXriYuLQ6fTsX37duEjTB4eHnh4eDj+7OnpycSJEwkICBAaV6n7rNT1wtg07+7du7FYLGRnZ3PgwAGXrH+0WCz8+te/5sKFC6jVauLj43nnnXdcclyWUtesRFsC2LRpEzdu3CA9PZ2JEydSXFxMeXm58LhK3WdQ7l4r0YeMx7YEyuQBKpWKXbt28fLLL2Oz2QgPD+eNN96goKBAaNzx+I4A5fouJeKKbseq/v5+8XNlkiRJkiRJkiTdQ+jRhpIkSZIkSZIkPZhMxiVJkiRJkiRJITIZlyRJkiRJkiSFyGRckiRJkiRJkhQik3FJkiRJkiRJUohMxiVJkiRJkiRJITIZlyRJkiRJkiSFyGRckiRJkiRJkhQik3FJkiRJkiRJUsj/ATUD5Klz7u3gAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "\u003cFigure size 936x216 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "N = 24\n",
        "(training_digits, training_labels,\n",
        " validation_digits, validation_labels) = dataset_to_numpy_util(training_dataset, validation_dataset, N)\n",
        "display_digits(training_digits, training_labels, training_labels, \"training digits and their labels\", N)\n",
        "display_digits(validation_digits[:N], validation_labels[:N], validation_labels[:N], \"validation digits and their labels\", N)\n",
        "font_digits, font_labels = create_digits_from_local_fonts(N)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "KIc0oqiD40HC"
      },
      "source": [
        "### Keras model: 3 convolutional layers, 2 dense layers\n",
        "If you are not sure what cross-entropy, dropout, softmax or batch-normalization mean, head here for a crash-course: [Tensorflow and deep learning without a PhD](https://github.com/GoogleCloudPlatform/tensorflow-without-a-phd/#featured-code-sample)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 708
        },
        "colab_type": "code",
        "id": "56y8UNFQIVwj",
        "outputId": "22cdb081-b945-4f74-c0da-199e2707e0c6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "image (Reshape)              (None, 28, 28, 1)         0         \n",
            "_________________________________________________________________\n",
            "conv2d_3 (Conv2D)            (None, 28, 28, 12)        108       \n",
            "_________________________________________________________________\n",
            "batch_normalization_4 (Batch (None, 28, 28, 12)        36        \n",
            "_________________________________________________________________\n",
            "activation_4 (Activation)    (None, 28, 28, 12)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_4 (Conv2D)            (None, 14, 14, 24)        10368     \n",
            "_________________________________________________________________\n",
            "batch_normalization_5 (Batch (None, 14, 14, 24)        72        \n",
            "_________________________________________________________________\n",
            "activation_5 (Activation)    (None, 14, 14, 24)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_5 (Conv2D)            (None, 7, 7, 32)          27648     \n",
            "_________________________________________________________________\n",
            "batch_normalization_6 (Batch (None, 7, 7, 32)          96        \n",
            "_________________________________________________________________\n",
            "activation_6 (Activation)    (None, 7, 7, 32)          0         \n",
            "_________________________________________________________________\n",
            "flatten_1 (Flatten)          (None, 1568)              0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 200)               313600    \n",
            "_________________________________________________________________\n",
            "batch_normalization_7 (Batch (None, 200)               600       \n",
            "_________________________________________________________________\n",
            "activation_7 (Activation)    (None, 200)               0         \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 200)               0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 10)                2010      \n",
            "=================================================================\n",
            "Total params: 354,538\n",
            "Trainable params: 354,002\n",
            "Non-trainable params: 536\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# This model trains to 99.4% accuracy in 10 epochs (with a batch size of 64)  \n",
        "\n",
        "def make_model():\n",
        "    model = tf.keras.Sequential(\n",
        "      [\n",
        "        tf.keras.layers.Reshape(input_shape=(28*28,), target_shape=(28, 28, 1), name=\"image\"),\n",
        "\n",
        "        tf.keras.layers.Conv2D(filters=12, kernel_size=3, padding='same', use_bias=False), # no bias necessary before batch norm\n",
        "        tf.keras.layers.BatchNormalization(scale=False, center=True), # no batch norm scaling necessary before \"relu\"\n",
        "        tf.keras.layers.Activation('relu'), # activation after batch norm\n",
        "\n",
        "        tf.keras.layers.Conv2D(filters=24, kernel_size=6, padding='same', use_bias=False, strides=2),\n",
        "        tf.keras.layers.BatchNormalization(scale=False, center=True),\n",
        "        tf.keras.layers.Activation('relu'),\n",
        "\n",
        "        tf.keras.layers.Conv2D(filters=32, kernel_size=6, padding='same', use_bias=False, strides=2),\n",
        "        tf.keras.layers.BatchNormalization(scale=False, center=True),\n",
        "        tf.keras.layers.Activation('relu'),\n",
        "\n",
        "        tf.keras.layers.Flatten(),\n",
        "        tf.keras.layers.Dense(200, use_bias=False),\n",
        "        tf.keras.layers.BatchNormalization(scale=False, center=True),\n",
        "        tf.keras.layers.Activation('relu'),\n",
        "        tf.keras.layers.Dropout(0.4), # Dropout on dense layer only\n",
        "\n",
        "        tf.keras.layers.Dense(10, activation='softmax')\n",
        "      ])\n",
        "\n",
        "    model.compile(optimizer='adam', # learning rate will be set by LearningRateScheduler\n",
        "                  loss='categorical_crossentropy',\n",
        "                  metrics=['accuracy'])\n",
        "    return model\n",
        "    \n",
        "with strategy.scope():\n",
        "    model = make_model()\n",
        "\n",
        "# print model layers\n",
        "model.summary()\n",
        "\n",
        "# set up learning rate decay\n",
        "lr_decay = tf.keras.callbacks.LearningRateScheduler(\n",
        "    lambda epoch: LEARNING_RATE * LEARNING_RATE_EXP_DECAY**epoch,\n",
        "    verbose=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CuhDh8ao8VyB"
      },
      "source": [
        "### Train and validate the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 760
        },
        "colab_type": "code",
        "id": "TTwH_P-ZJ_xx",
        "outputId": "7c6178a4-6bc7-49df-ca95-370e6d6c21d6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Steps per epoch:  117\n",
            "\n",
            "Epoch 00001: LearningRateScheduler reducing learning rate to 0.01.\n",
            "Epoch 1/10\n",
            "117/117 [==============================] - 1s 12ms/step - loss: 0.1829 - accuracy: 0.9450 - lr: 0.0100\n",
            "\n",
            "Epoch 00002: LearningRateScheduler reducing learning rate to 0.006999999999999999.\n",
            "Epoch 2/10\n",
            "117/117 [==============================] - 1s 11ms/step - loss: 0.0501 - accuracy: 0.9851 - lr: 0.0070\n",
            "\n",
            "Epoch 00003: LearningRateScheduler reducing learning rate to 0.0049.\n",
            "Epoch 3/10\n",
            "117/117 [==============================] - 1s 11ms/step - loss: 0.0331 - accuracy: 0.9898 - lr: 0.0049\n",
            "\n",
            "Epoch 00004: LearningRateScheduler reducing learning rate to 0.003429999999999999.\n",
            "Epoch 4/10\n",
            "117/117 [==============================] - 1s 11ms/step - loss: 0.0242 - accuracy: 0.9923 - lr: 0.0034\n",
            "\n",
            "Epoch 00005: LearningRateScheduler reducing learning rate to 0.0024009999999999995.\n",
            "Epoch 5/10\n",
            "117/117 [==============================] - 1s 10ms/step - loss: 0.0180 - accuracy: 0.9946 - lr: 0.0024\n",
            "\n",
            "Epoch 00006: LearningRateScheduler reducing learning rate to 0.0016806999999999994.\n",
            "Epoch 6/10\n",
            "117/117 [==============================] - 1s 11ms/step - loss: 0.0134 - accuracy: 0.9959 - lr: 0.0017\n",
            "\n",
            "Epoch 00007: LearningRateScheduler reducing learning rate to 0.0011764899999999997.\n",
            "Epoch 7/10\n",
            "117/117 [==============================] - 1s 11ms/step - loss: 0.0100 - accuracy: 0.9973 - lr: 0.0012\n",
            "\n",
            "Epoch 00008: LearningRateScheduler reducing learning rate to 0.0008235429999999996.\n",
            "Epoch 8/10\n",
            "117/117 [==============================] - 1s 10ms/step - loss: 0.0085 - accuracy: 0.9975 - lr: 8.2354e-04\n",
            "\n",
            "Epoch 00009: LearningRateScheduler reducing learning rate to 0.0005764800999999997.\n",
            "Epoch 9/10\n",
            "117/117 [==============================] - 1s 10ms/step - loss: 0.0071 - accuracy: 0.9982 - lr: 5.7648e-04\n",
            "\n",
            "Epoch 00010: LearningRateScheduler reducing learning rate to 0.0004035360699999998.\n",
            "Epoch 10/10\n",
            "117/117 [==============================] - 1s 10ms/step - loss: 0.0066 - accuracy: 0.9983 - lr: 4.0354e-04\n",
            "1/1 [==============================] - 0s 2ms/step - loss: 0.0188 - accuracy: 0.9945\n",
            "Validation accuracy:  0.9944999814033508\n"
          ]
        }
      ],
      "source": [
        "EPOCHS = 10\n",
        "steps_per_epoch = 60000//BATCH_SIZE  # 60,000 items in this dataset\n",
        "print(\"Steps per epoch: \", steps_per_epoch)\n",
        "  \n",
        "history = model.fit(training_dataset,\n",
        "                    steps_per_epoch=steps_per_epoch, epochs=EPOCHS,\n",
        "                    callbacks=[lr_decay])\n",
        "\n",
        "final_stats = model.evaluate(validation_dataset, steps=1)\n",
        "print(\"Validation accuracy: \", final_stats[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9jFVovcUUVs1"
      },
      "source": [
        "### Visualize predictions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "w12OId8Mz7dF"
      },
      "outputs": [],
      "source": [
        "# recognize digits from local fonts\n",
        "probabilities = model.predict(font_digits, steps=1)\n",
        "predicted_labels = np.argmax(probabilities, axis=1)\n",
        "display_digits(font_digits, predicted_labels, font_labels, \"predictions from local fonts (bad predictions in red)\", N)\n",
        "\n",
        "# recognize validation digits\n",
        "probabilities = model.predict(validation_digits, steps=1)\n",
        "predicted_labels = np.argmax(probabilities, axis=1)\n",
        "display_top_unrecognized(validation_digits, predicted_labels, validation_labels, N, 7)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5tzVi39ShrEL"
      },
      "source": [
        "## Deploy the trained model to AI Platform model serving\n",
        "\n",
        "Push your trained model to production on AI Platform for a serverless, autoscaled, REST API experience.\n",
        "\n",
        "You will need a GCS (Google Cloud Storage) bucket and a GCP project for this.\n",
        "Models deployed on AI Platform autoscale to zero if not used. There will be no AI Platform charges after you are done testing.\n",
        "Google Cloud Storage incurs charges. Empty the bucket after deployment if you want to avoid these. Once the model is deployed, the bucket is not useful anymore."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3Y3ztMY_toCP"
      },
      "source": [
        "### Configuration"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "iAZAn7yIhqAS"
      },
      "outputs": [],
      "source": [
        "PROJECT = \"ml-writers\" #@param {type:\"string\"}\n",
        "BUCKET = \"gs://your-bucket\"  #@param {type:\"string\", default:\"jddj\"}\n",
        "NEW_MODEL = True #@param {type:\"boolean\"}\n",
        "MODEL_NAME = \"mnist_test\" #@param {type:\"string\"}\n",
        "MODEL_VERSION = \"v1\" #@param {type:\"string\"}\n",
        "\n",
        "assert PROJECT, 'For this part, you need a GCP project. Head to http://console.cloud.google.com/ and create one.'\n",
        "assert re.search(r'gs://.+', BUCKET), 'For this part, you need a GCS bucket. Head to http://console.cloud.google.com/storage and create one.'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GxQTtjmdIbmN"
      },
      "source": [
        "### Export the model for serving from AI Platform"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "GOgh7Kb7SzzG"
      },
      "outputs": [],
      "source": [
        "# Wrap the model so that we can add a serving function\n",
        "class ExportModel(tf.keras.Model):\n",
        "  def __init__(self, model):\n",
        "    super().__init__(self)\n",
        "    self.model = model\n",
        "\n",
        "  # The serving function performig data pre- and post-processing.\n",
        "  # Pre-processing:  images are received in uint8 format converted\n",
        "  #                  to float32 before being sent to through the model.\n",
        "  # Post-processing: the Keras model outputs digit probabilities. We want\n",
        "  #                  the detected digits. An additional tf.argmax is needed.\n",
        "  # @tf.function turns the code in this function into a Tensorflow graph that\n",
        "  # can be exported. This way, the model itself, as well as its pre- and post-\n",
        "  # processing steps are exported in the SavedModel and deployed in a single step.\n",
        "  @tf.function(input_signature=[tf.TensorSpec([None, 28*28], dtype=tf.uint8)])\n",
        "  def my_serve(self, images):\n",
        "    images = tf.cast(images, tf.float32)/255   # pre-processing\n",
        "    probabilities = self.model(images)          # prediction from model\n",
        "    classes = tf.argmax(probabilities, axis=-1) # post-processing\n",
        "    return {'digits': classes}\n",
        "    \n",
        "# Must copy the model from TPU to CPU to be able to compose them.\n",
        "restored_model = make_model()\n",
        "restored_model.set_weights(model.get_weights()) # this copies the weights from TPU, does nothing on GPU\n",
        "\n",
        "# create the ExportModel and export it to the Tensorflow standard SavedModel format\n",
        "serving_model = ExportModel(restored_model)\n",
        "export_path = os.path.join(BUCKET, 'keras_export', str(time.time()))\n",
        "tf.keras.backend.set_learning_phase(0) # inference only\n",
        "tf.saved_model.save(serving_model, export_path, signatures={'serving_default': serving_model.my_serve})\n",
        "\n",
        "print(\"Model exported to: \", export_path)\n",
        "\n",
        "# Note: in Tensorflow 2.0, it will also be possible to\n",
        "# export to the SavedModel format using model.save():\n",
        "# serving_model.save(export_path, save_format='tf')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Zm7cpCRQC8-w"
      },
      "outputs": [],
      "source": [
        "# saved_model_cli: a useful tool for troubleshooting SavedModels (the tool is part of the Tensorflow installation)\n",
        "!saved_model_cli show --dir {export_path}\n",
        "!saved_model_cli show --dir {export_path} --tag_set serve\n",
        "!saved_model_cli show --dir {export_path} --tag_set serve --signature_def serving_default\n",
        "# A note on naming:\n",
        "# The \"serve\" tag set (i.e. serving functionality) is the only one exported by tf.saved_model.save\n",
        "# All the other names are defined by the user in the fllowing lines of code:\n",
        "#      def myserve(self, images):\n",
        "#                        ******\n",
        "#        return {'digits': classes}\n",
        "#                 ******\n",
        "#      tf.saved_model.save(..., signatures={'serving_default': serving_model.myserve})\n",
        "#                                            ***************"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zy3T3zk0u2J0"
      },
      "source": [
        "### Deploy the model\n",
        "This uses the command-line interface. You can do the same thing through the AI Platform UI at https://console.cloud.google.com/mlengine/models\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "nGv3ITiGLPL3"
      },
      "outputs": [],
      "source": [
        "# Create the model\n",
        "if NEW_MODEL:\n",
        "  !gcloud ai-platform models create {MODEL_NAME} --project={PROJECT} --regions=us-central1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "o3QtUowtOAL-"
      },
      "outputs": [],
      "source": [
        "# Create a version of this model (you can add --async at the end of the line to make this call non blocking)\n",
        "# Additional config flags are available: https://cloud.google.com/ml-engine/reference/rest/v1/projects.models.versions\n",
        "# You can also deploy a model that is stored locally by providing a --staging-bucket=... parameter\n",
        "!echo \"Deployment takes a couple of minutes. You can watch your deployment here: https://console.cloud.google.com/mlengine/models/{MODEL_NAME}\"\n",
        "!gcloud ai-platform versions create {MODEL_VERSION} --model={MODEL_NAME} --origin={export_path} --project={PROJECT} --runtime-version=1.14 --python-version=3.5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jE-k1Zn6kU2Z"
      },
      "source": [
        "### Test the deployed model\n",
        "Your model is now available as a REST API. Let us try to call it. The cells below use the \"gcloud ml-engine\"\n",
        "command line tool but any tool that can send a JSON payload to a REST endpoint will work."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "zZCt0Ke2QDer"
      },
      "outputs": [],
      "source": [
        "# prepare digits to send to online prediction endpoint\n",
        "digits_float32 = np.concatenate((font_digits, validation_digits[:100-N])) # pixel values in [0.0, 1.0] float range\n",
        "digits_uint8 = np.round(digits_float32*255).astype(np.uint8) # pixel values in [0, 255] int range\n",
        "labels = np.concatenate((font_labels, validation_labels[:100-N]))\n",
        "with open(\"digits.json\", \"w\") as f:\n",
        "  for digit in digits_uint8:\n",
        "    # the format for AI Platform online predictions is: one JSON object per line\n",
        "    data = json.dumps({\"images\": digit.tolist()})  # \"images\" because that was the name you gave this parametr in the serving funtion my_serve\n",
        "    f.write(data+'\\n')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "n6PqhQ8RQ8bp"
      },
      "outputs": [],
      "source": [
        "# Request online predictions from deployed model (REST API) using the \"gcloud ml-engine\" command line.\n",
        "predictions = !gcloud ai-platform predict --model={MODEL_NAME} --json-instances digits.json --project={PROJECT} --version {MODEL_VERSION}\n",
        "print(predictions)\n",
        "\n",
        "predictions = np.stack([json.loads(p) for p in predictions[2:]]) # first elemet is the name of the output layer: drop it, parse the rest\n",
        "display_top_unrecognized(digits_float32, predictions, labels, N, 100//N)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2a5cGsSTEBQD"
      },
      "source": [
        "## What's next\n",
        "\n",
        "* Learn about [Cloud TPUs](https://cloud.google.com/tpu/docs) that Google designed and optimized specifically to speed up and scale up ML workloads for training and inference and to enable ML engineers and researchers to iterate more quickly.\n",
        "* Explore the range of [Cloud TPU tutorials and Colabs](https://cloud.google.com/tpu/docs/tutorials) to find other examples that can be used when implementing your ML project.\n",
        "\n",
        "On Google Cloud Platform, in addition to GPUs and TPUs available on pre-configured [deep learning VMs](https://cloud.google.com/deep-learning-vm/),  you will find [AutoML](https://cloud.google.com/automl/)*(beta)* for training custom models without writing code and [Cloud ML Engine](https://cloud.google.com/ml-engine/docs/) which will allows you to run parallel trainings and hyperparameter tuning of your custom models on powerful distributed hardware.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SVY1pBg5ydH-"
      },
      "source": [
        "## License"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hleIN5-pcr0N"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "author: Martin Gorner\u003cbr\u003e\n",
        "twitter: @martin_gorner\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "Copyright 2019 Google LLC\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "you may not use this file except in compliance with the License.\n",
        "You may obtain a copy of the License at\n",
        "\n",
        "    http://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "Unless required by applicable law or agreed to in writing, software\n",
        "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "See the License for the specific language governing permissions and\n",
        "limitations under the License.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "This is not an official Google product but sample code provided for an educational purpose\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "collapsed_sections": [
        "TBsuwHGAv7w4"
      ],
      "name": "Keras_MNIST_TPU.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
